2024-02-10T05:41:00.259 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-10T05:41:00.260 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --name yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi --description orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} --archive /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555100 --verbose -- /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555100/orig.config.yaml 2024-02-10T05:41:00.289 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T05:41:00.365 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555100 branch: main description: orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} email: yweinste@redhat.com first_in_suite: false job_id: '7555100' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 1610 sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 sleep_before_teardown: 0 subset: 111/120000 suite: orch suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 targets: smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC3oADVnAjG9kBHaL+3tviPsiRUn9yOC3HIBlbR+MQTyM6R4VnjyOZCqmh8FQYIbZewyjMrk2exkNpwpA0mXdiQ= smithi176.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNfbPxrNPOC1vHKDdw9nlAz0VjN2se4rgoRXzZtrlHaH9TPrXjn45WGqndy6jrY117N+i5Wfh1kTVwNbJPw2M90= tasks: - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-09_19:44:16 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-10T05:41:00.365 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa; will attempt to use it 2024-02-10T05:41:00.366 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks 2024-02-10T05:41:00.366 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-10T05:41:00.368 INFO:teuthology.task.internal:Checking packages... 2024-02-10T05:41:00.395 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90' 2024-02-10T05:41:00.395 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T05:41:00.395 INFO:teuthology.packaging:ref: None 2024-02-10T05:41:00.396 INFO:teuthology.packaging:tag: None 2024-02-10T05:41:00.396 INFO:teuthology.packaging:branch: main 2024-02-10T05:41:00.396 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:41:00.396 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-02-10T05:41:00.651 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1361.gf8e4e331 2024-02-10T05:41:00.653 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-10T05:41:00.661 INFO:teuthology.task.internal:no buildpackages task found 2024-02-10T05:41:00.661 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-10T05:41:00.669 INFO:teuthology.task.internal:Saving configuration 2024-02-10T05:41:00.681 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-10T05:41:00.688 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T05:41:00.711 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555100', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:36:10.721307', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC3oADVnAjG9kBHaL+3tviPsiRUn9yOC3HIBlbR+MQTyM6R4VnjyOZCqmh8FQYIbZewyjMrk2exkNpwpA0mXdiQ='} 2024-02-10T05:41:00.733 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555100', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:36:10.719784', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNfbPxrNPOC1vHKDdw9nlAz0VjN2se4rgoRXzZtrlHaH9TPrXjn45WGqndy6jrY117N+i5Wfh1kTVwNbJPw2M90='} 2024-02-10T05:41:00.733 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-10T05:41:00.739 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-02-10T05:41:00.739 INFO:teuthology.task.internal:roles: ubuntu@smithi176.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-02-10T05:41:00.739 INFO:teuthology.run_tasks:Running task console_log... 2024-02-10T05:41:00.799 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fba328c84c0>, signals=[15]) 2024-02-10T05:41:00.799 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-10T05:41:00.805 INFO:teuthology.task.internal:Opening connections... 2024-02-10T05:41:00.806 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2024-02-10T05:41:00.808 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:41:00.914 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi176.front.sepia.ceph.com 2024-02-10T05:41:00.915 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:41:01.005 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-10T05:41:01.012 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2024-02-10T05:41:01.030 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2024-02-10T05:41:01.030 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2024-02-10T05:41:01.090 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="CentOS Stream" 2024-02-10T05:41:01.091 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="9" 2024-02-10T05:41:01.091 INFO:teuthology.orchestra.run.smithi064.stdout:ID="centos" 2024-02-10T05:41:01.091 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE="rhel fedora" 2024-02-10T05:41:01.091 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="9" 2024-02-10T05:41:01.091 INFO:teuthology.orchestra.run.smithi064.stdout:PLATFORM_ID="platform:el9" 2024-02-10T05:41:01.091 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-10T05:41:01.091 INFO:teuthology.orchestra.run.smithi064.stdout:ANSI_COLOR="0;31" 2024-02-10T05:41:01.091 INFO:teuthology.orchestra.run.smithi064.stdout:LOGO="fedora-logo-icon" 2024-02-10T05:41:01.091 INFO:teuthology.orchestra.run.smithi064.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-10T05:41:01.091 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://centos.org/" 2024-02-10T05:41:01.092 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T05:41:01.092 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-10T05:41:01.092 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T05:41:01.092 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2024-02-10T05:41:01.119 DEBUG:teuthology.orchestra.run.smithi176:> uname -m 2024-02-10T05:41:01.137 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64 2024-02-10T05:41:01.137 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/os-release 2024-02-10T05:41:01.194 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="CentOS Stream" 2024-02-10T05:41:01.194 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="9" 2024-02-10T05:41:01.194 INFO:teuthology.orchestra.run.smithi176.stdout:ID="centos" 2024-02-10T05:41:01.195 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE="rhel fedora" 2024-02-10T05:41:01.195 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="9" 2024-02-10T05:41:01.195 INFO:teuthology.orchestra.run.smithi176.stdout:PLATFORM_ID="platform:el9" 2024-02-10T05:41:01.195 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-10T05:41:01.195 INFO:teuthology.orchestra.run.smithi176.stdout:ANSI_COLOR="0;31" 2024-02-10T05:41:01.195 INFO:teuthology.orchestra.run.smithi176.stdout:LOGO="fedora-logo-icon" 2024-02-10T05:41:01.195 INFO:teuthology.orchestra.run.smithi176.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-10T05:41:01.195 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://centos.org/" 2024-02-10T05:41:01.195 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T05:41:01.195 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-10T05:41:01.195 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T05:41:01.196 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2024-02-10T05:41:01.219 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-10T05:41:01.231 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-10T05:41:01.242 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-10T05:41:01.242 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2024-02-10T05:41:01.245 DEBUG:teuthology.orchestra.run.smithi176:> test '!' -e /home/ubuntu/cephtest 2024-02-10T05:41:01.261 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-10T05:41:01.269 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-10T05:41:01.269 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2024-02-10T05:41:01.304 DEBUG:teuthology.orchestra.run.smithi176:> test -z $(ls -A /var/lib/ceph) 2024-02-10T05:41:01.371 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-10T05:41:01.427 INFO:teuthology.run_tasks:Running task kernel... 2024-02-10T05:41:01.445 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-10T05:41:01.445 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-10T05:41:01.445 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T05:41:01.446 DEBUG:teuthology.orchestra.run.smithi176:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T05:41:01.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:41:01.462 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-02-10T05:41:01.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:41:01.463 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-02-10T05:41:01.520 INFO:teuthology.orchestra.run.smithi176.stdout:5.14.0-361.el9.x86_64 2024-02-10T05:41:01.521 INFO:teuthology.task.kernel:Running kernel on smithi176: 5.14.0-361.el9.x86_64 2024-02-10T05:41:01.521 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum install -y kernel 2024-02-10T05:41:01.523 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-361.el9.x86_64 2024-02-10T05:41:01.523 INFO:teuthology.task.kernel:Running kernel on smithi064: 5.14.0-361.el9.x86_64 2024-02-10T05:41:01.523 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum install -y kernel 2024-02-10T05:41:03.527 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-02-10T05:41:03.598 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-02-10T05:41:04.993 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - BaseOS 7.7 MB/s | 2.0 MB 00:00 2024-02-10T05:41:05.207 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-02-10T05:41:06.734 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 6.2 MB/s | 8.0 MB 00:01 2024-02-10T05:41:08.005 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - BaseOS 3.3 MB/s | 8.0 MB 00:02 2024-02-10T05:41:13.693 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 3.3 MB/s | 19 MB 00:05 2024-02-10T05:41:13.947 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - AppStream 3.9 MB/s | 19 MB 00:04 2024-02-10T05:41:17.877 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 11 MB/s | 5.9 MB 00:00 2024-02-10T05:41:19.253 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 15 kB 00:00 2024-02-10T05:41:20.587 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 17 MB/s | 20 MB 00:01 2024-02-10T05:41:21.706 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - CRB 1.4 MB/s | 5.9 MB 00:04 2024-02-10T05:41:23.415 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 15 kB 00:00 2024-02-10T05:41:25.049 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 14 MB/s | 20 MB 00:01 2024-02-10T05:41:25.637 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 50 kB/s | 1.7 kB 00:00 2024-02-10T05:41:27.936 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-10T05:41:27.936 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-10T05:41:28.037 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-02-10T05:41:28.044 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-02-10T05:41:28.044 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-02-10T05:41:28.044 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-02-10T05:41:28.044 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-02-10T05:41:28.044 INFO:teuthology.orchestra.run.smithi064.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-10T05:41:28.044 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-02-10T05:41:28.045 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-10T05:41:28.045 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-10T05:41:28.045 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-10T05:41:28.045 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:41:28.045 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-02-10T05:41:28.045 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2024-02-10T05:41:28.045 INFO:teuthology.orchestra.run.smithi064.stdout:Install 4 Packages 2024-02-10T05:41:28.045 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:41:28.046 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 99 M 2024-02-10T05:41:28.046 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 122 M 2024-02-10T05:41:28.046 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-02-10T05:41:28.488 INFO:teuthology.orchestra.run.smithi064.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 14 MB/s | 6.0 MB 00:00 2024-02-10T05:41:29.580 INFO:teuthology.orchestra.run.smithi064.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 14 MB/s | 21 MB 00:01 2024-02-10T05:41:29.924 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-02-10T05:41:29.939 INFO:teuthology.orchestra.run.smithi064.stdout:(3/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 21 MB/s | 39 MB 00:01 2024-02-10T05:41:30.564 INFO:teuthology.orchestra.run.smithi064.stdout:(4/4): kernel-modules-core-5.14.0-419.el9.x86_6 16 MB/s | 33 MB 00:02 2024-02-10T05:41:30.567 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:41:30.567 INFO:teuthology.orchestra.run.smithi064.stdout:Total 39 MB/s | 99 MB 00:02 2024-02-10T05:41:30.568 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-02-10T05:41:31.016 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-02-10T05:41:31.017 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-02-10T05:41:32.098 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-02-10T05:41:32.099 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-02-10T05:41:32.189 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-10T05:41:32.189 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-10T05:41:32.278 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-02-10T05:41:32.294 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-10T05:41:32.294 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-02-10T05:41:32.294 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-10T05:41:32.294 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-02-10T05:41:32.295 INFO:teuthology.orchestra.run.smithi176.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-10T05:41:32.295 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-02-10T05:41:32.295 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-10T05:41:32.295 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-10T05:41:32.295 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-10T05:41:32.295 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-10T05:41:32.295 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-02-10T05:41:32.295 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-02-10T05:41:32.295 INFO:teuthology.orchestra.run.smithi176.stdout:Install 4 Packages 2024-02-10T05:41:32.295 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-10T05:41:32.296 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 99 M 2024-02-10T05:41:32.296 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 122 M 2024-02-10T05:41:32.296 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-02-10T05:41:32.767 INFO:teuthology.orchestra.run.smithi176.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 13 MB/s | 6.0 MB 00:00 2024-02-10T05:41:33.459 INFO:teuthology.orchestra.run.smithi176.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 18 MB/s | 21 MB 00:01 2024-02-10T05:41:34.242 INFO:teuthology.orchestra.run.smithi176.stdout:(3/4): kernel-modules-core-5.14.0-419.el9.x86_6 23 MB/s | 33 MB 00:01 2024-02-10T05:41:34.633 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-02-10T05:41:34.884 INFO:teuthology.orchestra.run.smithi176.stdout:(4/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 15 MB/s | 39 MB 00:02 2024-02-10T05:41:34.887 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:41:34.888 INFO:teuthology.orchestra.run.smithi176.stdout:Total 38 MB/s | 99 MB 00:02 2024-02-10T05:41:34.888 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-02-10T05:41:35.275 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-02-10T05:41:35.275 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-02-10T05:41:35.307 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:41:35.439 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:41:36.285 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-02-10T05:41:36.285 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-02-10T05:41:38.325 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:41:38.506 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:41:38.690 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-02-10T05:41:39.587 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:41:39.717 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:41:42.470 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:41:42.709 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:41:43.567 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:41:43.686 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:41:47.413 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:41:47.578 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:41:48.775 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:41:52.331 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:42:19.610 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:42:19.610 INFO:teuthology.orchestra.run.smithi064.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-10T05:42:19.610 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:42:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:42:20.089 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:42:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:42:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:42:20.090 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:42:20.543 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:42:20.543 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:42:20.543 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-02-10T05:42:20.543 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-10T05:42:20.543 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-10T05:42:20.543 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-10T05:42:20.543 INFO:teuthology.orchestra.run.smithi064.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-10T05:42:20.543 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:42:20.543 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-02-10T05:42:21.030 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:42:21.160 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:42:21.160 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:42:21.161 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-10T05:42:21.161 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T05:42:21.161 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T05:42:21.161 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-10T05:42:21.161 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi064.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T05:42:21.161 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:42:21.273 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:42:21.273 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:42:21.273 DEBUG:teuthology.orchestra.run.smithi064:> sudo rpm -qi grub2-tools 2024-02-10T05:42:21.311 INFO:teuthology.orchestra.run.smithi064.stdout:Name : grub2-tools 2024-02-10T05:42:21.311 INFO:teuthology.orchestra.run.smithi064.stdout:Epoch : 1 2024-02-10T05:42:21.311 INFO:teuthology.orchestra.run.smithi064.stdout:Version : 2.06 2024-02-10T05:42:21.311 INFO:teuthology.orchestra.run.smithi064.stdout:Release : 68.el9 2024-02-10T05:42:21.311 INFO:teuthology.orchestra.run.smithi064.stdout:Architecture: x86_64 2024-02-10T05:42:21.311 INFO:teuthology.orchestra.run.smithi064.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-10T05:42:21.311 INFO:teuthology.orchestra.run.smithi064.stdout:Group : Unspecified 2024-02-10T05:42:21.311 INFO:teuthology.orchestra.run.smithi064.stdout:Size : 8264254 2024-02-10T05:42:21.311 INFO:teuthology.orchestra.run.smithi064.stdout:License : GPLv3+ 2024-02-10T05:42:21.312 INFO:teuthology.orchestra.run.smithi064.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-10T05:42:21.312 INFO:teuthology.orchestra.run.smithi064.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-10T05:42:21.312 INFO:teuthology.orchestra.run.smithi064.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-10T05:42:21.312 INFO:teuthology.orchestra.run.smithi064.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-10T05:42:21.312 INFO:teuthology.orchestra.run.smithi064.stdout:Packager : builder@centos.org 2024-02-10T05:42:21.312 INFO:teuthology.orchestra.run.smithi064.stdout:Vendor : CentOS 2024-02-10T05:42:21.312 INFO:teuthology.orchestra.run.smithi064.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T05:42:21.312 INFO:teuthology.orchestra.run.smithi064.stdout:Summary : Support tools for GRUB. 2024-02-10T05:42:21.312 INFO:teuthology.orchestra.run.smithi064.stdout:Description : 2024-02-10T05:42:21.312 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:42:21.312 INFO:teuthology.orchestra.run.smithi064.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T05:42:21.312 INFO:teuthology.orchestra.run.smithi064.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T05:42:21.313 INFO:teuthology.orchestra.run.smithi064.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T05:42:21.313 INFO:teuthology.orchestra.run.smithi064.stdout:hardware devices. 2024-02-10T05:42:21.313 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:42:21.313 INFO:teuthology.orchestra.run.smithi064.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T05:42:21.314 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T05:42:21.314 INFO:teuthology.task.kernel:Updating grub on smithi064 to boot 5.14.0-419.el9.x86_64 2024-02-10T05:42:21.314 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T05:42:21.413 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:42:21.413 INFO:teuthology.orchestra.run.smithi176.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-10T05:42:21.413 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-10T05:42:21.483 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:42:21.871 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:42:21.871 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:42:21.872 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:42:21.872 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:42:21.984 INFO:teuthology.orchestra.run.smithi064.stderr:Generating grub configuration file ... 2024-02-10T05:42:22.324 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:42:22.325 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-10T05:42:22.325 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-02-10T05:42:22.325 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-10T05:42:22.325 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-10T05:42:22.325 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-10T05:42:22.325 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-10T05:42:22.325 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-10T05:42:22.325 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-02-10T05:42:22.801 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:42:22.934 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:42:22.934 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:42:22.934 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-10T05:42:22.934 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T05:42:22.935 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T05:42:22.935 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-02-10T05:42:22.935 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi176.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T05:42:22.935 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:42:23.043 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:42:23.044 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:42:23.044 DEBUG:teuthology.orchestra.run.smithi176:> sudo rpm -qi grub2-tools 2024-02-10T05:42:23.091 INFO:teuthology.orchestra.run.smithi176.stdout:Name : grub2-tools 2024-02-10T05:42:23.091 INFO:teuthology.orchestra.run.smithi176.stdout:Epoch : 1 2024-02-10T05:42:23.091 INFO:teuthology.orchestra.run.smithi176.stdout:Version : 2.06 2024-02-10T05:42:23.092 INFO:teuthology.orchestra.run.smithi176.stdout:Release : 68.el9 2024-02-10T05:42:23.092 INFO:teuthology.orchestra.run.smithi176.stdout:Architecture: x86_64 2024-02-10T05:42:23.092 INFO:teuthology.orchestra.run.smithi176.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-10T05:42:23.092 INFO:teuthology.orchestra.run.smithi176.stdout:Group : Unspecified 2024-02-10T05:42:23.092 INFO:teuthology.orchestra.run.smithi176.stdout:Size : 8264254 2024-02-10T05:42:23.092 INFO:teuthology.orchestra.run.smithi176.stdout:License : GPLv3+ 2024-02-10T05:42:23.092 INFO:teuthology.orchestra.run.smithi176.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-10T05:42:23.092 INFO:teuthology.orchestra.run.smithi176.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-10T05:42:23.092 INFO:teuthology.orchestra.run.smithi176.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-10T05:42:23.092 INFO:teuthology.orchestra.run.smithi176.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-10T05:42:23.092 INFO:teuthology.orchestra.run.smithi176.stdout:Packager : builder@centos.org 2024-02-10T05:42:23.092 INFO:teuthology.orchestra.run.smithi176.stdout:Vendor : CentOS 2024-02-10T05:42:23.092 INFO:teuthology.orchestra.run.smithi176.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T05:42:23.093 INFO:teuthology.orchestra.run.smithi176.stdout:Summary : Support tools for GRUB. 2024-02-10T05:42:23.093 INFO:teuthology.orchestra.run.smithi176.stdout:Description : 2024-02-10T05:42:23.093 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-10T05:42:23.093 INFO:teuthology.orchestra.run.smithi176.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T05:42:23.093 INFO:teuthology.orchestra.run.smithi176.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T05:42:23.093 INFO:teuthology.orchestra.run.smithi176.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T05:42:23.093 INFO:teuthology.orchestra.run.smithi176.stdout:hardware devices. 2024-02-10T05:42:23.093 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-10T05:42:23.093 INFO:teuthology.orchestra.run.smithi176.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T05:42:23.094 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T05:42:23.094 INFO:teuthology.task.kernel:Updating grub on smithi176 to boot 5.14.0-419.el9.x86_64 2024-02-10T05:42:23.094 DEBUG:teuthology.orchestra.run.smithi176:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T05:42:23.376 INFO:teuthology.orchestra.run.smithi064.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-10T05:42:23.414 INFO:teuthology.orchestra.run.smithi064.stderr:done 2024-02-10T05:42:23.416 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-02-10T05:42:23.431 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.eRPXUqX3rf 2024-02-10T05:42:23.431 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.eRPXUqX3rf 2024-02-10T05:42:23.503 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.eRPXUqX3rf 2024-02-10T05:42:23.641 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.eRPXUqX3rf is 6KB 2024-02-10T05:42:23.680 INFO:teuthology.orchestra.run.smithi176.stderr:Generating grub configuration file ... 2024-02-10T05:42:23.689 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.eRPXUqX3rf 2024-02-10T05:42:23.704 DEBUG:teuthology.orchestra.run.smithi064:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T05:42:23.774 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-10T05:42:23.774 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-10T05:42:23.774 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-10T05:42:23.774 INFO:teuthology.orchestra.run.smithi064.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-10T05:42:23.775 DEBUG:teuthology.orchestra.run.smithi064:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-10T05:42:23.921 DEBUG:teuthology.orchestra.run.smithi064:> sudo shutdown -r now 2024-02-10T05:42:25.033 INFO:teuthology.orchestra.run.smithi176.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-10T05:42:25.071 INFO:teuthology.orchestra.run.smithi176.stderr:done 2024-02-10T05:42:25.073 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-02-10T05:42:25.093 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.Sj39SNaV1z 2024-02-10T05:42:25.093 DEBUG:teuthology.orchestra.run.smithi176:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Sj39SNaV1z 2024-02-10T05:42:25.162 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.Sj39SNaV1z 2024-02-10T05:42:25.293 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.Sj39SNaV1z is 6KB 2024-02-10T05:42:25.340 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.Sj39SNaV1z 2024-02-10T05:42:25.354 DEBUG:teuthology.orchestra.run.smithi176:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T05:42:25.419 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-10T05:42:25.420 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-10T05:42:25.420 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-10T05:42:25.420 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-10T05:42:25.421 DEBUG:teuthology.orchestra.run.smithi176:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-10T05:42:25.572 DEBUG:teuthology.orchestra.run.smithi176:> sudo shutdown -r now 2024-02-10T05:42:53.952 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-10T05:42:53.952 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-02-10T05:42:53.953 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:42:55.576 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-02-10T05:42:55.576 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-02-10T05:42:55.577 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:43:54.013 DEBUG:teuthology.orchestra.remote:timed out 2024-02-10T05:43:55.580 DEBUG:teuthology.orchestra.remote:timed out 2024-02-10T05:44:03.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-02-10T05:44:03.021 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:44:03.426 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-02-10T05:44:03.886 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-02-10T05:44:03.886 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-419.el9.x86_64"... 2024-02-10T05:44:03.887 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-02-10T05:44:03.900 INFO:teuthology.orchestra.run.smithi064.stdout:5.14.0-419.el9.x86_64 2024-02-10T05:44:03.901 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-10T05:44:03.901 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T05:44:03.901 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T05:44:04.581 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-02-10T05:44:04.582 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:44:04.902 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-10T05:44:04.902 DEBUG:teuthology.orchestra.run.smithi064:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T05:44:04.963 INFO:teuthology.orchestra.run.smithi064.stdout:ttyS1 2024-02-10T05:44:04.992 DEBUG:teuthology.parallel:result is None 2024-02-10T05:44:05.023 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-02-10T05:44:05.473 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-02-10T05:44:05.473 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-419.el9.x86_64"... 2024-02-10T05:44:05.473 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-02-10T05:44:05.487 INFO:teuthology.orchestra.run.smithi176.stdout:5.14.0-419.el9.x86_64 2024-02-10T05:44:05.487 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-10T05:44:05.488 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T05:44:05.488 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T05:44:06.489 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-02-10T05:44:06.489 DEBUG:teuthology.orchestra.run.smithi176:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T05:44:06.541 INFO:teuthology.orchestra.run.smithi176.stdout:ttyS1 2024-02-10T05:44:06.570 DEBUG:teuthology.parallel:result is None 2024-02-10T05:44:06.570 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-10T05:44:06.577 INFO:teuthology.task.internal:Creating test directory... 2024-02-10T05:44:06.577 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T05:44:06.580 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T05:44:06.594 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-10T05:44:06.600 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-10T05:44:06.608 INFO:teuthology.task.internal:Creating archive directory... 2024-02-10T05:44:06.608 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T05:44:06.636 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T05:44:06.657 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-10T05:44:06.663 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-10T05:44:06.664 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T05:44:06.691 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T05:44:06.720 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:44:06.724 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:44:06.733 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:44:06.737 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:44:06.739 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-10T05:44:06.746 INFO:teuthology.task.internal:Configuring sudo... 2024-02-10T05:44:06.746 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T05:44:06.780 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T05:44:06.807 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-10T05:44:06.816 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-10T05:44:06.817 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T05:44:06.850 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T05:44:06.865 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:44:06.945 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:44:07.003 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:44:07.084 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:44:07.136 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T05:44:07.136 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T05:44:07.206 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:44:07.245 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:44:07.292 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:44:07.367 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:44:07.408 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-10T05:44:07.408 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T05:44:07.472 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2024-02-10T05:44:07.474 DEBUG:teuthology.orchestra.run.smithi176:> sudo service rsyslog restart 2024-02-10T05:44:07.538 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T05:44:07.563 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T05:44:08.048 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-10T05:44:08.056 INFO:teuthology.task.internal:Starting timer... 2024-02-10T05:44:08.056 INFO:teuthology.run_tasks:Running task pcp... 2024-02-10T05:44:08.068 INFO:teuthology.run_tasks:Running task selinux... 2024-02-10T05:44:08.077 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-10T05:44:08.077 DEBUG:teuthology.orchestra.run.smithi064:> sudo service auditd rotate 2024-02-10T05:44:08.136 INFO:teuthology.orchestra.run.smithi064.stdout:Rotating logs: 2024-02-10T05:44:08.138 DEBUG:teuthology.orchestra.run.smithi176:> sudo service auditd rotate 2024-02-10T05:44:08.245 INFO:teuthology.orchestra.run.smithi176.stdout:Rotating logs: 2024-02-10T05:44:08.249 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-10T05:44:08.250 DEBUG:teuthology.orchestra.run.smithi064:> /usr/sbin/getenforce 2024-02-10T05:44:08.273 INFO:teuthology.orchestra.run.smithi064.stdout:Permissive 2024-02-10T05:44:08.273 DEBUG:teuthology.orchestra.run.smithi176:> /usr/sbin/getenforce 2024-02-10T05:44:08.326 INFO:teuthology.orchestra.run.smithi176.stdout:Permissive 2024-02-10T05:44:08.326 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi064.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi176.front.sepia.ceph.com': 'permissive'} 2024-02-10T05:44:08.326 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T05:44:08.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:44:08.354 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T05:44:08.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:44:08.401 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-10T05:44:08.401 DEBUG:teuthology.orchestra.run.smithi064:> sudo /usr/sbin/setenforce permissive 2024-02-10T05:44:08.451 DEBUG:teuthology.orchestra.run.smithi176:> sudo /usr/sbin/setenforce permissive 2024-02-10T05:44:08.687 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-10T05:44:08.697 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-10T05:44:08.713 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-10T05:44:08.714 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi064.front.sepia.ceph.com,smithi176.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-10T05:48:19.750 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi064.front.sepia.ceph.com'), Remote(name='ubuntu@smithi176.front.sepia.ceph.com')] 2024-02-10T05:48:19.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-02-10T05:48:19.753 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:48:19.846 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-02-10T05:48:19.926 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-02-10T05:48:19.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-02-10T05:48:19.927 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:48:20.033 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-02-10T05:48:20.113 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-02-10T05:48:20.114 INFO:teuthology.run_tasks:Running task clock... 2024-02-10T05:48:20.125 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-10T05:48:20.126 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T05:48:20.126 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T05:48:20.128 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T05:48:20.128 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T05:48:20.157 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T05:48:20.174 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T05:48:20.184 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T05:48:20.200 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T05:48:20.203 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: ntpd: command not found 2024-02-10T05:48:20.216 INFO:teuthology.orchestra.run.smithi064.stdout:506 Cannot talk to daemon 2024-02-10T05:48:20.231 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: ntpd: command not found 2024-02-10T05:48:20.233 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T05:48:20.243 INFO:teuthology.orchestra.run.smithi176.stdout:506 Cannot talk to daemon 2024-02-10T05:48:20.249 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T05:48:20.259 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T05:48:20.276 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T05:48:20.304 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-02-10T05:48:20.307 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T05:48:20.307 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-02-10T05:48:20.308 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:48:20.308 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:48:20.308 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:48:20.308 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:48:20.334 INFO:teuthology.orchestra.run.smithi176.stderr:bash: line 1: ntpq: command not found 2024-02-10T05:48:20.337 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T05:48:20.337 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2024-02-10T05:48:20.337 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:48:20.337 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:48:20.337 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:48:20.338 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:48:20.338 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-02-10T05:48:20.349 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-02-10T05:48:20.349 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T05:48:20.349 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T05:48:20.364 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T05:48:20.364 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2024-02-10T05:48:20.418 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T05:48:20.419 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:48:20.419 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-02-10T05:48:20.419 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:48:20.419 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:48:20.419 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-02-10 05:48:18.760371160 +0000 2024-02-10T05:48:20.419 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-02-10 05:47:57.580781217 +0000 2024-02-10T05:48:20.419 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-02-10 05:47:57.580781217 +0000 2024-02-10T05:48:20.419 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-02-10 05:47:57.580781217 +0000 2024-02-10T05:48:20.419 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T05:48:20.484 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-02-10T05:48:20.485 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-02-10T05:48:20.485 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000245505 s, 2.1 MB/s 2024-02-10T05:48:20.486 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T05:48:20.541 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2024-02-10T05:48:20.595 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T05:48:20.595 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:48:20.595 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-02-10T05:48:20.595 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:48:20.595 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:48:20.595 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-02-10 05:48:18.760371160 +0000 2024-02-10T05:48:20.595 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-02-10 05:47:57.943774189 +0000 2024-02-10T05:48:20.595 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-02-10 05:47:57.943774189 +0000 2024-02-10T05:48:20.595 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-02-10 05:47:57.943774189 +0000 2024-02-10T05:48:20.596 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T05:48:20.662 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-02-10T05:48:20.662 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-02-10T05:48:20.662 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000190427 s, 2.7 MB/s 2024-02-10T05:48:20.662 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T05:48:20.717 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2024-02-10T05:48:20.771 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T05:48:20.771 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:48:20.771 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-02-10T05:48:20.771 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:48:20.771 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:48:20.771 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-02-10 05:48:18.760371160 +0000 2024-02-10T05:48:20.771 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-02-10 05:47:58.274767781 +0000 2024-02-10T05:48:20.772 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-02-10 05:47:58.274767781 +0000 2024-02-10T05:48:20.772 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-02-10 05:47:58.274767781 +0000 2024-02-10T05:48:20.772 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T05:48:20.836 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-02-10T05:48:20.836 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-02-10T05:48:20.836 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000206817 s, 2.5 MB/s 2024-02-10T05:48:20.837 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T05:48:20.892 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2024-02-10T05:48:20.947 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T05:48:20.947 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:48:20.947 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-02-10T05:48:20.947 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:48:20.948 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:48:20.948 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-02-10 05:48:18.760371160 +0000 2024-02-10T05:48:20.948 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-02-10 05:47:58.601761450 +0000 2024-02-10T05:48:20.948 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-02-10 05:47:58.601761450 +0000 2024-02-10T05:48:20.948 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-02-10 05:47:58.601761450 +0000 2024-02-10T05:48:20.948 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T05:48:21.011 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-02-10T05:48:21.011 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-02-10T05:48:21.012 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000175513 s, 2.9 MB/s 2024-02-10T05:48:21.013 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T05:48:21.068 DEBUG:teuthology.orchestra.run.smithi064:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-02-10T05:48:21.211 INFO:teuthology.orchestra.run.smithi064.stdout:loop 2024-02-10T05:48:21.212 INFO:tasks.nvme_loop:Connecting nvme_loop smithi064:/dev/vg_nvme/lv_1... 2024-02-10T05:48:21.212 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-02-10T05:48:21.250 INFO:teuthology.orchestra.run.smithi064.stdout:1 2024-02-10T05:48:21.287 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/vg_nvme/lv_11 2024-02-10T05:48:21.317 INFO:tasks.nvme_loop:Connecting nvme_loop smithi064:/dev/vg_nvme/lv_2... 2024-02-10T05:48:21.317 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-02-10T05:48:21.400 INFO:teuthology.orchestra.run.smithi064.stdout:1 2024-02-10T05:48:21.438 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/vg_nvme/lv_21 2024-02-10T05:48:21.466 INFO:tasks.nvme_loop:Connecting nvme_loop smithi064:/dev/vg_nvme/lv_3... 2024-02-10T05:48:21.466 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-02-10T05:48:21.544 INFO:teuthology.orchestra.run.smithi064.stdout:1 2024-02-10T05:48:21.581 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/vg_nvme/lv_31 2024-02-10T05:48:21.608 INFO:tasks.nvme_loop:Connecting nvme_loop smithi064:/dev/vg_nvme/lv_4... 2024-02-10T05:48:21.608 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-02-10T05:48:21.686 INFO:teuthology.orchestra.run.smithi064.stdout:1 2024-02-10T05:48:21.723 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/vg_nvme/lv_41 2024-02-10T05:48:21.751 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T05:48:21.751 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T05:48:21.806 DEBUG:teuthology.orchestra.run.smithi064:> sudo nvme list 2024-02-10T05:48:21.871 INFO:teuthology.orchestra.run.smithi064.stdout:Node SN Model Namespace Usage Format FW Rev 2024-02-10T05:48:21.871 INFO:teuthology.orchestra.run.smithi064.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-02-10T05:48:21.871 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/nvme0n1 CVFT534400C2400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-02-10T05:48:21.871 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/nvme1n1 96fb1ebcbe27fbb0f429 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-10T05:48:21.871 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/nvme2n1 84314c54f9fa6806e2d9 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-10T05:48:21.872 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/nvme3n1 476a0469b662288393e0 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-10T05:48:21.872 INFO:teuthology.orchestra.run.smithi064.stdout:/dev/nvme4n1 0368b02efd5ab14ae36a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-10T05:48:21.873 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-02-10T05:48:21.873 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T05:48:21.873 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/scratch_devs 2024-02-10T05:48:21.940 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-10T05:48:21.940 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T05:48:21.958 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T05:48:21.959 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_1 2024-02-10T05:48:22.017 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T05:48:22.017 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:48:22.017 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-02-10T05:48:22.017 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:48:22.017 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:48:22.017 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-02-10 05:48:18.735062367 +0000 2024-02-10T05:48:22.017 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-02-10 05:47:55.026266047 +0000 2024-02-10T05:48:22.017 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-02-10 05:47:55.026266047 +0000 2024-02-10T05:48:22.017 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-02-10 05:47:55.026266047 +0000 2024-02-10T05:48:22.017 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T05:48:22.088 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-02-10T05:48:22.088 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-02-10T05:48:22.088 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000263816 s, 1.9 MB/s 2024-02-10T05:48:22.089 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T05:48:22.147 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_2 2024-02-10T05:48:22.204 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T05:48:22.204 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:48:22.204 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-02-10T05:48:22.204 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:48:22.204 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:48:22.204 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-02-10 05:48:18.735062367 +0000 2024-02-10T05:48:22.204 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-02-10 05:47:55.386262955 +0000 2024-02-10T05:48:22.204 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-02-10 05:47:55.386262955 +0000 2024-02-10T05:48:22.204 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-02-10 05:47:55.386262955 +0000 2024-02-10T05:48:22.204 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T05:48:22.272 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-02-10T05:48:22.273 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-02-10T05:48:22.273 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000239751 s, 2.1 MB/s 2024-02-10T05:48:22.274 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T05:48:22.331 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_3 2024-02-10T05:48:22.387 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T05:48:22.387 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:48:22.388 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-02-10T05:48:22.388 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:48:22.388 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:48:22.388 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-02-10 05:48:18.735062367 +0000 2024-02-10T05:48:22.388 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-02-10 05:47:55.729260008 +0000 2024-02-10T05:48:22.388 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-02-10 05:47:55.729260008 +0000 2024-02-10T05:48:22.388 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-02-10 05:47:55.729260008 +0000 2024-02-10T05:48:22.388 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T05:48:22.454 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-02-10T05:48:22.454 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-02-10T05:48:22.455 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000213203 s, 2.4 MB/s 2024-02-10T05:48:22.455 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T05:48:22.511 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_4 2024-02-10T05:48:22.566 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T05:48:22.566 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:48:22.566 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-02-10T05:48:22.566 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:48:22.567 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:48:22.567 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-02-10 05:48:18.735062367 +0000 2024-02-10T05:48:22.567 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-02-10 05:47:56.083256967 +0000 2024-02-10T05:48:22.567 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-02-10 05:47:56.083256967 +0000 2024-02-10T05:48:22.567 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-02-10 05:47:56.083256967 +0000 2024-02-10T05:48:22.567 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T05:48:22.632 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-02-10T05:48:22.632 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-02-10T05:48:22.633 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000163289 s, 3.1 MB/s 2024-02-10T05:48:22.633 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T05:48:22.689 DEBUG:teuthology.orchestra.run.smithi176:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-02-10T05:48:22.858 INFO:teuthology.orchestra.run.smithi176.stdout:loop 2024-02-10T05:48:22.859 INFO:tasks.nvme_loop:Connecting nvme_loop smithi176:/dev/vg_nvme/lv_1... 2024-02-10T05:48:22.859 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-02-10T05:48:22.897 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-02-10T05:48:22.934 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/vg_nvme/lv_11 2024-02-10T05:48:22.963 INFO:tasks.nvme_loop:Connecting nvme_loop smithi176:/dev/vg_nvme/lv_2... 2024-02-10T05:48:22.963 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-02-10T05:48:23.049 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-02-10T05:48:23.088 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/vg_nvme/lv_21 2024-02-10T05:48:23.118 INFO:tasks.nvme_loop:Connecting nvme_loop smithi176:/dev/vg_nvme/lv_3... 2024-02-10T05:48:23.119 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-02-10T05:48:23.197 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-02-10T05:48:23.236 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/vg_nvme/lv_31 2024-02-10T05:48:23.274 INFO:tasks.nvme_loop:Connecting nvme_loop smithi176:/dev/vg_nvme/lv_4... 2024-02-10T05:48:23.274 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-02-10T05:48:23.353 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-02-10T05:48:23.393 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/vg_nvme/lv_41 2024-02-10T05:48:23.422 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-10T05:48:23.423 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T05:48:23.477 DEBUG:teuthology.orchestra.run.smithi176:> sudo nvme list 2024-02-10T05:48:23.546 INFO:teuthology.orchestra.run.smithi176.stdout:Node SN Model Namespace Usage Format FW Rev 2024-02-10T05:48:23.546 INFO:teuthology.orchestra.run.smithi176.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-02-10T05:48:23.546 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/nvme0n1 CVFT62330099400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-02-10T05:48:23.546 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/nvme1n1 d5f950076274d28fd035 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-10T05:48:23.546 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/nvme2n1 f55c4e9ca67a7fbf5c07 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-10T05:48:23.546 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/nvme3n1 766529302a64b7756be0 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-10T05:48:23.546 INFO:teuthology.orchestra.run.smithi176.stdout:/dev/nvme4n1 359591fff069553d4dab Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-10T05:48:23.547 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-02-10T05:48:23.548 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-10T05:48:23.548 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/scratch_devs 2024-02-10T05:48:23.616 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-10T05:48:23.715 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'global': {'mon election default strategy': 3}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'} 2024-02-10T05:48:23.715 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:48:23.716 INFO:tasks.cephadm:Cluster fsid is 00e86dbe-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:48:23.716 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-10T05:48:23.716 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.64', 'mon.c': '[v2:172.21.15.64:3301,v1:172.21.15.64:6790]', 'mon.b': '172.21.15.176'} 2024-02-10T05:48:23.716 INFO:tasks.cephadm:First mon is mon.a on smithi064 2024-02-10T05:48:23.716 INFO:tasks.cephadm:First mgr is y 2024-02-10T05:48:23.716 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-10T05:48:23.716 DEBUG:teuthology.orchestra.run.smithi064:> sudo hostname $(hostname -s) 2024-02-10T05:48:23.748 DEBUG:teuthology.orchestra.run.smithi176:> sudo hostname $(hostname -s) 2024-02-10T05:48:23.775 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-02-10T05:48:23.776 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:48:23.935 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/76638/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1361-g290212a1', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1361.g290212a1'}, 'url': 'https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-02-09 18:34:21.832169', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'main', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-02-10T05:48:24.091 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref main, sha1 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:48:24.093 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/x86_64/flavors/default/cephadm 2024-02-10T05:48:24.093 DEBUG:teuthology.orchestra.run.smithi064:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T05:48:24.567 INFO:teuthology.orchestra.run.smithi064.stdout:-rw-r--r--. 1 ubuntu ubuntu 766809 Feb 10 05:48 /home/ubuntu/cephtest/cephadm 2024-02-10T05:48:24.567 DEBUG:teuthology.orchestra.run.smithi176:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T05:48:25.071 INFO:teuthology.orchestra.run.smithi176.stdout:-rw-r--r--. 1 ubuntu ubuntu 766809 Feb 10 05:48 /home/ubuntu/cephtest/cephadm 2024-02-10T05:48:25.071 DEBUG:teuthology.orchestra.run.smithi064:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T05:48:25.091 DEBUG:teuthology.orchestra.run.smithi176:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T05:48:25.120 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 on all hosts... 2024-02-10T05:48:25.120 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 pull 2024-02-10T05:48:25.133 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 pull 2024-02-10T05:48:25.995 INFO:teuthology.orchestra.run.smithi176.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90... 2024-02-10T05:48:26.006 INFO:teuthology.orchestra.run.smithi064.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90... 2024-02-10T05:49:09.071 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-10T05:49:09.072 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_version": "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)", 2024-02-10T05:49:09.072 INFO:teuthology.orchestra.run.smithi064.stdout: "image_id": "c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6", 2024-02-10T05:49:09.072 INFO:teuthology.orchestra.run.smithi064.stdout: "repo_digests": [ 2024-02-10T05:49:09.072 INFO:teuthology.orchestra.run.smithi064.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58" 2024-02-10T05:49:09.072 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-02-10T05:49:09.072 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-10T05:49:17.301 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-02-10T05:49:17.302 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_version": "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)", 2024-02-10T05:49:17.302 INFO:teuthology.orchestra.run.smithi176.stdout: "image_id": "c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6", 2024-02-10T05:49:17.302 INFO:teuthology.orchestra.run.smithi176.stdout: "repo_digests": [ 2024-02-10T05:49:17.302 INFO:teuthology.orchestra.run.smithi176.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58" 2024-02-10T05:49:17.302 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-02-10T05:49:17.302 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-02-10T05:49:17.325 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph 2024-02-10T05:49:17.361 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /etc/ceph 2024-02-10T05:49:17.391 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /etc/ceph 2024-02-10T05:49:17.430 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 777 /etc/ceph 2024-02-10T05:49:17.458 INFO:tasks.cephadm:Writing seed config... 2024-02-10T05:49:17.459 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-10T05:49:17.459 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-10T05:49:17.460 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-02-10T05:49:17.460 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-02-10T05:49:17.460 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-10T05:49:17.460 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-10T05:49:17.460 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-10T05:49:17.460 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-10T05:49:17.461 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-10T05:49:17.461 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-02-10T05:49:17.462 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T05:49:17.462 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-10T05:49:17.488 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 00e86dbe-c7d8-11ee-95b8-87774f69a715 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-02-10T05:49:17.488 DEBUG:teuthology.orchestra.run.smithi064:mon.a> sudo journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@mon.a.service 2024-02-10T05:49:17.530 DEBUG:teuthology.orchestra.run.smithi064:mgr.y> sudo journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@mgr.y.service 2024-02-10T05:49:17.574 INFO:tasks.cephadm:Bootstrapping... 2024-02-10T05:49:17.574 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -v bootstrap --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.64 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:49:17.772 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:49:17.773 INFO:teuthology.orchestra.run.smithi064.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', '-v', 'bootstrap', '--fsid', '00e86dbe-c7d8-11ee-95b8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.64', '--skip-admin-label'] 2024-02-10T05:49:17.800 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-10T05:49:17.800 INFO:teuthology.orchestra.run.smithi064.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-02-10T05:49:17.801 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying podman|docker is present... 2024-02-10T05:49:17.825 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-10T05:49:17.826 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying lvm2 is present... 2024-02-10T05:49:17.826 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying time synchronization is in place... 2024-02-10T05:49:17.834 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-10T05:49:17.834 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T05:49:17.842 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-10T05:49:17.842 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-02-10T05:49:17.850 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout enabled 2024-02-10T05:49:17.857 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout active 2024-02-10T05:49:17.857 INFO:teuthology.orchestra.run.smithi064.stdout:Unit chronyd.service is enabled and running 2024-02-10T05:49:17.857 INFO:teuthology.orchestra.run.smithi064.stdout:Repeating the final host check... 2024-02-10T05:49:17.882 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-10T05:49:17.882 INFO:teuthology.orchestra.run.smithi064.stdout:podman (/bin/podman) version 4.9.3 is present 2024-02-10T05:49:17.882 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl is present 2024-02-10T05:49:17.882 INFO:teuthology.orchestra.run.smithi064.stdout:lvcreate is present 2024-02-10T05:49:17.888 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-10T05:49:17.888 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T05:49:17.916 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-10T05:49:17.916 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-02-10T05:49:17.923 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout enabled 2024-02-10T05:49:17.930 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout active 2024-02-10T05:49:17.930 INFO:teuthology.orchestra.run.smithi064.stdout:Unit chronyd.service is enabled and running 2024-02-10T05:49:17.931 INFO:teuthology.orchestra.run.smithi064.stdout:Host looks OK 2024-02-10T05:49:17.931 INFO:teuthology.orchestra.run.smithi064.stdout:Cluster fsid: 00e86dbe-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:49:17.931 INFO:teuthology.orchestra.run.smithi064.stdout:Acquiring lock 139803977013856 on /run/cephadm/00e86dbe-c7d8-11ee-95b8-87774f69a715.lock 2024-02-10T05:49:17.931 INFO:teuthology.orchestra.run.smithi064.stdout:Lock 139803977013856 acquired on /run/cephadm/00e86dbe-c7d8-11ee-95b8-87774f69a715.lock 2024-02-10T05:49:17.931 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying IP 172.21.15.64 port 3300 ... 2024-02-10T05:49:17.932 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying IP 172.21.15.64 port 6789 ... 2024-02-10T05:49:17.932 INFO:teuthology.orchestra.run.smithi064.stdout:Base mon IP(s) is [172.21.15.64:3300, 172.21.15.64:6789], mon addrv is [v2:172.21.15.64:3300,v1:172.21.15.64:6789] 2024-02-10T05:49:17.936 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.64 metric 100 2024-02-10T05:49:17.937 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.64 metric 100 2024-02-10T05:49:17.940 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-02-10T05:49:17.940 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-10T05:49:17.940 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1797sec hoplimit 64 pref medium 2024-02-10T05:49:17.944 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-10T05:49:17.944 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-02-10T05:49:17.944 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-10T05:49:17.944 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-10T05:49:17.944 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:feb6:f8df/64 scope link 2024-02-10T05:49:17.944 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-10T05:49:17.945 INFO:teuthology.orchestra.run.smithi064.stdout:Mon IP `172.21.15.64` is in CIDR network `172.21.0.0/20` 2024-02-10T05:49:17.945 INFO:teuthology.orchestra.run.smithi064.stdout:Mon IP `172.21.15.64` is in CIDR network `172.21.0.0/20` 2024-02-10T05:49:17.945 INFO:teuthology.orchestra.run.smithi064.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-10T05:49:17.946 INFO:teuthology.orchestra.run.smithi064.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-10T05:49:17.947 INFO:teuthology.orchestra.run.smithi064.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90... 2024-02-10T05:49:18.443 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stdout c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 2024-02-10T05:49:18.443 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90... 2024-02-10T05:49:18.443 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Getting image source signatures 2024-02-10T05:49:18.443 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Copying blob sha256:5fe0e97a267976e17c7dcb240cf9455472de6876101ae78af4bdf182158b6d8d 2024-02-10T05:49:18.443 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Copying blob sha256:58df4e03ba066accf3a7e07426bab5f1c867e27a139801c642bdafb42b9a2d9a 2024-02-10T05:49:18.443 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Copying config sha256:c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 2024-02-10T05:49:18.443 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Writing manifest to image destination 2024-02-10T05:49:19.349 INFO:teuthology.orchestra.run.smithi064.stdout:ceph: stdout ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-10T05:49:19.349 INFO:teuthology.orchestra.run.smithi064.stdout:Ceph version: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-10T05:49:19.349 INFO:teuthology.orchestra.run.smithi064.stdout:Extracting ceph user uid/gid from container image... 2024-02-10T05:49:20.213 INFO:teuthology.orchestra.run.smithi064.stdout:stat: stdout 167 167 2024-02-10T05:49:20.213 INFO:teuthology.orchestra.run.smithi064.stdout:Creating initial keys... 2024-02-10T05:49:20.969 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph-authtool: stdout AQBgDsdltrM3HxAAjteyLuvwQUlV/lPJPiKgYw== 2024-02-10T05:49:21.858 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph-authtool: stdout AQBhDsdlGUcbEhAAiGDlVpHeKk6JGQJoHCSO8g== 2024-02-10T05:49:22.721 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph-authtool: stdout AQBiDsdl/WBuCBAAl3U4Su8fLl3pB8dYrrfSbA== 2024-02-10T05:49:22.722 INFO:teuthology.orchestra.run.smithi064.stdout:Creating initial monmap... 2024-02-10T05:49:23.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T05:49:23.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-02-10T05:49:23.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 00e86dbe-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:49:23.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T05:49:23.636 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool for a [v2:172.21.15.64:3300,v1:172.21.15.64:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T05:49:23.636 INFO:teuthology.orchestra.run.smithi064.stdout:setting min_mon_release = quincy 2024-02-10T05:49:23.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: set fsid to 00e86dbe-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:49:23.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T05:49:23.636 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:49:23.636 INFO:teuthology.orchestra.run.smithi064.stdout:Creating mon... 2024-02-10T05:49:24.541 INFO:teuthology.orchestra.run.smithi064.stdout:create mon.a on 2024-02-10T05:49:24.917 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-10T05:49:25.098 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715.target. 2024-02-10T05:49:25.098 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715.target. 2024-02-10T05:49:25.331 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@mon.a 2024-02-10T05:49:25.331 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Failed to reset failed state of unit ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@mon.a.service: Unit ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@mon.a.service not loaded. 2024-02-10T05:49:25.498 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715.target.wants/ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@.service. 2024-02-10T05:49:26.015 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-10T05:49:26.015 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout disabled 2024-02-10T05:49:26.022 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-10T05:49:26.022 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-02-10T05:49:26.022 INFO:teuthology.orchestra.run.smithi064.stdout:firewalld.service is not enabled 2024-02-10T05:49:26.022 INFO:teuthology.orchestra.run.smithi064.stdout:Not possible to enable service . firewalld.service is not available 2024-02-10T05:49:26.022 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mon to start... 2024-02-10T05:49:26.022 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mon... 2024-02-10T05:49:27.053 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout cluster: 2024-02-10T05:49:27.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout id: 00e86dbe-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:49:27.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-02-10T05:49:27.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-02-10T05:49:27.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout services: 2024-02-10T05:49:27.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.49857s) 2024-02-10T05:49:27.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-02-10T05:49:27.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-02-10T05:49:27.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-02-10T05:49:27.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout data: 2024-02-10T05:49:27.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-02-10T05:49:27.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-02-10T05:49:27.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-02-10T05:49:27.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout pgs: 2024-02-10T05:49:27.054 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-02-10T05:49:27.055 INFO:teuthology.orchestra.run.smithi064.stdout:mon is available 2024-02-10T05:49:27.055 INFO:teuthology.orchestra.run.smithi064.stdout:Assimilating anything we can from ceph.conf... 2024-02-10T05:49:28.001 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-02-10T05:49:28.001 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [global] 2024-02-10T05:49:28.002 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout fsid = 00e86dbe-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:49:28.002 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.64:3300,v1:172.21.15.64:6789] 2024-02-10T05:49:28.002 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-10T05:49:28.002 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-10T05:49:28.002 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-10T05:49:28.002 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-10T05:49:28.002 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-02-10T05:49:28.002 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-10T05:49:28.002 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = False 2024-02-10T05:49:28.002 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-10T05:49:28.002 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-02-10T05:49:28.002 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [osd] 2024-02-10T05:49:28.002 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-10T05:49:28.003 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-10T05:49:28.003 INFO:teuthology.orchestra.run.smithi064.stdout:Generating new minimal ceph.conf... 2024-02-10T05:49:29.023 INFO:teuthology.orchestra.run.smithi064.stdout:Restarting the monitor... 2024-02-10T05:49:29.274 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:29 smithi064 systemd[1]: Stopping Ceph mon.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:49:29.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:29 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-a[24803]: 2024-02-10T05:49:29.273+0000 7fc80fbf5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:49:29.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:29 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-a[24803]: 2024-02-10T05:49:29.273+0000 7fc80fbf5700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-10T05:49:29.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:29 smithi064 podman[25054]: 2024-02-10 05:49:29.478442868 +0000 UTC m=+0.293751585 container died b5f3fd9d68b4b892a2f1d10de0c25133155f36b858019dc2ec1b5eaa01d82fbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-02-10T05:49:30.068 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:29 smithi064 podman[25054]: 2024-02-10 05:49:29.790072278 +0000 UTC m=+0.605380994 container cleanup b5f3fd9d68b4b892a2f1d10de0c25133155f36b858019dc2ec1b5eaa01d82fbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.build-date=20240131) 2024-02-10T05:49:30.068 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:29 smithi064 bash[25054]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-a 2024-02-10T05:49:30.068 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:29 smithi064 podman[25067]: 2024-02-10 05:49:29.957533214 +0000 UTC m=+0.479341366 container remove b5f3fd9d68b4b892a2f1d10de0c25133155f36b858019dc2ec1b5eaa01d82fbc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-02-10T05:49:30.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 systemd[1]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@mon.a.service: Deactivated successfully. 2024-02-10T05:49:30.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 systemd[1]: Stopped Ceph mon.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:49:30.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 systemd[1]: Starting Ceph mon.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:49:30.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 podman[25166]: 2024-02-10 05:49:30.193394541 +0000 UTC m=+0.017370997 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:49:30.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 podman[25166]: 2024-02-10 05:49:30.291567027 +0000 UTC m=+0.115543484 container create 22a0019eee341f33a7f0065792d0be56b53f5fae6d028ec29d471d635912da54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-a, RELEASE=main-290212a, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T05:49:30.572 INFO:teuthology.orchestra.run.smithi064.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-02-10T05:49:30.678 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 podman[25166]: 2024-02-10 05:49:30.493882212 +0000 UTC m=+0.317858668 container init 22a0019eee341f33a7f0065792d0be56b53f5fae6d028ec29d471d635912da54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-a, ceph=True, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:49:30.678 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 podman[25166]: 2024-02-10 05:49:30.496665938 +0000 UTC m=+0.320642405 container start 22a0019eee341f33a7f0065792d0be56b53f5fae6d028ec29d471d635912da54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-a, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-02-10T05:49:30.678 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T05:49:30.678 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev), process ceph-mon, pid 2 2024-02-10T05:49:30.678 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: pidfile_write: ignore empty --pid-file 2024-02-10T05:49:30.678 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: load: jerasure load: lrc 2024-02-10T05:49:30.678 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: RocksDB version: 7.9.2 2024-02-10T05:49:30.678 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Git sha 0 2024-02-10T05:49:30.679 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Compile date 2024-02-09 17:35:33 2024-02-10T05:49:30.679 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: DB SUMMARY 2024-02-10T05:49:30.679 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: DB Session ID: JIVCCT1KBAG5RYITMTFS 2024-02-10T05:49:30.679 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: CURRENT file: CURRENT 2024-02-10T05:49:30.679 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T05:49:30.679 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-02-10T05:49:30.679 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-02-10T05:49:30.679 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84627 ; 2024-02-10T05:49:30.679 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.error_if_exists: 0 2024-02-10T05:49:30.679 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.create_if_missing: 0 2024-02-10T05:49:30.679 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.paranoid_checks: 1 2024-02-10T05:49:30.679 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-10T05:49:30.679 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T05:49:30.679 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-10T05:49:30.679 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.env: 0x556e24bdeae0 2024-02-10T05:49:30.680 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.fs: PosixFileSystem 2024-02-10T05:49:30.680 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.info_log: 0x556e26c338c0 2024-02-10T05:49:30.680 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T05:49:30.680 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.statistics: (nil) 2024-02-10T05:49:30.681 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.use_fsync: 0 2024-02-10T05:49:30.681 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_log_file_size: 0 2024-02-10T05:49:30.681 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T05:49:30.681 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T05:49:30.682 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T05:49:30.682 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T05:49:30.682 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.allow_fallocate: 1 2024-02-10T05:49:30.682 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T05:49:30.682 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T05:49:30.682 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.use_direct_reads: 0 2024-02-10T05:49:30.682 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T05:49:30.682 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T05:49:30.682 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.db_log_dir: 2024-02-10T05:49:30.682 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.wal_dir: 2024-02-10T05:49:30.682 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T05:49:30.682 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T05:49:30.682 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T05:49:30.683 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T05:49:30.683 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T05:49:30.683 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T05:49:30.683 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T05:49:30.683 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T05:49:30.683 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.write_buffer_manager: 0x556e26c07400 2024-02-10T05:49:30.683 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T05:49:30.683 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T05:49:30.683 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T05:49:30.683 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T05:49:30.684 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T05:49:30.684 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T05:49:30.684 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T05:49:30.684 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T05:49:30.684 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.unordered_write: 0 2024-02-10T05:49:30.684 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T05:49:30.684 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T05:49:30.684 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T05:49:30.684 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T05:49:30.684 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.row_cache: None 2024-02-10T05:49:30.684 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.wal_filter: None 2024-02-10T05:49:30.684 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T05:49:30.684 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T05:49:30.685 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.two_write_queues: 0 2024-02-10T05:49:30.685 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T05:49:30.685 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.wal_compression: 0 2024-02-10T05:49:30.685 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.atomic_flush: 0 2024-02-10T05:49:30.685 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T05:49:30.685 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T05:49:30.685 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T05:49:30.685 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.log_readahead_size: 0 2024-02-10T05:49:30.685 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T05:49:30.685 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T05:49:30.685 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T05:49:30.685 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T05:49:30.685 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T05:49:30.685 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T05:49:30.686 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-10T05:49:30.686 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_background_jobs: 2 2024-02-10T05:49:30.686 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_background_compactions: -1 2024-02-10T05:49:30.686 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_subcompactions: 1 2024-02-10T05:49:30.686 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T05:49:30.686 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T05:49:30.686 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T05:49:30.686 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T05:49:30.686 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T05:49:30.686 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T05:49:30.686 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T05:49:30.686 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T05:49:30.686 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_open_files: -1 2024-02-10T05:49:30.686 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T05:49:30.687 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T05:49:30.687 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T05:49:30.687 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T05:49:30.687 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_background_flushes: -1 2024-02-10T05:49:30.687 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Compression algorithms supported: 2024-02-10T05:49:30.687 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T05:49:30.687 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: kZSTD supported: 0 2024-02-10T05:49:30.687 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: kXpressCompression supported: 0 2024-02-10T05:49:30.687 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T05:49:30.687 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: kZlibCompression supported: 1 2024-02-10T05:49:30.687 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: kSnappyCompression supported: 1 2024-02-10T05:49:30.687 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: kLZ4Compression supported: 1 2024-02-10T05:49:30.687 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: kBZip2Compression supported: 0 2024-02-10T05:49:30.687 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T05:49:30.688 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-10T05:49:30.688 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-02-10T05:49:30.688 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-10T05:49:30.688 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T05:49:30.688 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.merge_operator: 2024-02-10T05:49:30.688 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.compaction_filter: None 2024-02-10T05:49:30.688 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T05:49:30.688 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T05:49:30.688 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T05:49:30.688 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T05:49:30.688 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556e26c339c0) 2024-02-10T05:49:30.688 INFO:journalctl@ceph.mon.a.smithi064.stdout: cache_index_and_filter_blocks: 1 2024-02-10T05:49:30.688 INFO:journalctl@ceph.mon.a.smithi064.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T05:49:30.689 INFO:journalctl@ceph.mon.a.smithi064.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T05:49:30.689 INFO:journalctl@ceph.mon.a.smithi064.stdout: pin_top_level_index_and_filter: 1 2024-02-10T05:49:30.689 INFO:journalctl@ceph.mon.a.smithi064.stdout: index_type: 0 2024-02-10T05:49:30.689 INFO:journalctl@ceph.mon.a.smithi064.stdout: data_block_index_type: 0 2024-02-10T05:49:30.689 INFO:journalctl@ceph.mon.a.smithi064.stdout: index_shortening: 1 2024-02-10T05:49:30.689 INFO:journalctl@ceph.mon.a.smithi064.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T05:49:30.689 INFO:journalctl@ceph.mon.a.smithi064.stdout: checksum: 4 2024-02-10T05:49:30.689 INFO:journalctl@ceph.mon.a.smithi064.stdout: no_block_cache: 0 2024-02-10T05:49:30.689 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache: 0x556e26c37090 2024-02-10T05:49:30.689 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache_name: BinnedLRUCache 2024-02-10T05:49:30.689 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache_options: 2024-02-10T05:49:30.689 INFO:journalctl@ceph.mon.a.smithi064.stdout: capacity : 536870912 2024-02-10T05:49:30.689 INFO:journalctl@ceph.mon.a.smithi064.stdout: num_shard_bits : 4 2024-02-10T05:49:30.689 INFO:journalctl@ceph.mon.a.smithi064.stdout: strict_capacity_limit : 0 2024-02-10T05:49:30.690 INFO:journalctl@ceph.mon.a.smithi064.stdout: high_pri_pool_ratio: 0.000 2024-02-10T05:49:30.690 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache_compressed: (nil) 2024-02-10T05:49:30.690 INFO:journalctl@ceph.mon.a.smithi064.stdout: persistent_cache: (nil) 2024-02-10T05:49:30.690 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_size: 4096 2024-02-10T05:49:30.690 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_size_deviation: 10 2024-02-10T05:49:30.690 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_restart_interval: 16 2024-02-10T05:49:30.690 INFO:journalctl@ceph.mon.a.smithi064.stdout: index_block_restart_interval: 1 2024-02-10T05:49:30.690 INFO:journalctl@ceph.mon.a.smithi064.stdout: metadata_block_size: 4096 2024-02-10T05:49:30.690 INFO:journalctl@ceph.mon.a.smithi064.stdout: partition_filters: 0 2024-02-10T05:49:30.690 INFO:journalctl@ceph.mon.a.smithi064.stdout: use_delta_encoding: 1 2024-02-10T05:49:30.691 INFO:journalctl@ceph.mon.a.smithi064.stdout: filter_policy: bloomfilter 2024-02-10T05:49:30.691 INFO:journalctl@ceph.mon.a.smithi064.stdout: whole_key_filtering: 1 2024-02-10T05:49:30.691 INFO:journalctl@ceph.mon.a.smithi064.stdout: verify_compression: 0 2024-02-10T05:49:30.691 INFO:journalctl@ceph.mon.a.smithi064.stdout: read_amp_bytes_per_bit: 0 2024-02-10T05:49:30.691 INFO:journalctl@ceph.mon.a.smithi064.stdout: format_version: 5 2024-02-10T05:49:30.691 INFO:journalctl@ceph.mon.a.smithi064.stdout: enable_index_compression: 1 2024-02-10T05:49:30.691 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_align: 0 2024-02-10T05:49:30.691 INFO:journalctl@ceph.mon.a.smithi064.stdout: max_auto_readahead_size: 262144 2024-02-10T05:49:30.691 INFO:journalctl@ceph.mon.a.smithi064.stdout: prepopulate_block_cache: 0 2024-02-10T05:49:30.691 INFO:journalctl@ceph.mon.a.smithi064.stdout: initial_auto_readahead_size: 8192 2024-02-10T05:49:30.691 INFO:journalctl@ceph.mon.a.smithi064.stdout: num_file_reads_for_auto_readahead: 2 2024-02-10T05:49:30.691 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T05:49:30.691 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T05:49:30.691 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.compression: NoCompression 2024-02-10T05:49:30.692 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T05:49:30.692 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T05:49:30.692 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T05:49:30.692 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.num_levels: 7 2024-02-10T05:49:30.692 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T05:49:30.692 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T05:49:30.692 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T05:49:30.692 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T05:49:30.692 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T05:49:30.692 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T05:49:30.692 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T05:49:30.692 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:49:30.692 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T05:49:30.692 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T05:49:30.693 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-10T05:49:30.693 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-10T05:49:30.693 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T05:49:30.693 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T05:49:30.693 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T05:49:30.693 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T05:49:30.693 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:49:30.693 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-10T05:49:30.693 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T05:49:30.693 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T05:49:30.693 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-10T05:49:30.693 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T05:49:30.693 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T05:49:30.693 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T05:49:30.694 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T05:49:30.694 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T05:49:30.694 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T05:49:30.694 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T05:49:30.694 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T05:49:30.694 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T05:49:30.694 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T05:49:30.694 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T05:49:30.694 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T05:49:30.694 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T05:49:30.694 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T05:49:30.694 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T05:49:30.694 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T05:49:30.694 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T05:49:30.694 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-10T05:49:30.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.arena_block_size: 1048576 2024-02-10T05:49:30.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T05:49:30.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T05:49:30.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T05:49:30.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T05:49:30.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T05:49:30.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T05:49:30.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T05:49:30.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T05:49:30.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T05:49:30.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T05:49:30.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T05:49:30.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T05:49:30.695 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T05:49:30.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.table_properties_collectors: 2024-02-10T05:49:30.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.inplace_update_support: 0 2024-02-10T05:49:30.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T05:49:30.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T05:49:30.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T05:49:30.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T05:49:30.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.bloom_locality: 0 2024-02-10T05:49:30.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.max_successive_merges: 0 2024-02-10T05:49:30.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T05:49:30.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T05:49:30.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T05:49:30.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T05:49:30.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.ttl: 2592000 2024-02-10T05:49:30.696 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T05:49:30.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-10T05:49:30.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-10T05:49:30.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.enable_blob_files: false 2024-02-10T05:49:30.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.min_blob_size: 0 2024-02-10T05:49:30.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T05:49:30.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T05:49:30.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T05:49:30.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T05:49:30.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-10T05:49:30.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-10T05:49:30.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.blob_file_starting_level: 0 2024-02-10T05:49:30.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-10T05:49:30.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-02-10T05:49:30.697 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-02-10T05:49:30.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b02fd721-0f5a-455b-b0d8-13a3856a6d70 2024-02-10T05:49:30.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544170525581, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-02-10T05:49:30.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-02-10T05:49:30.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544170526847, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81252, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 79439, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 73748, "raw_average_value_size": 349, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707544170, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b02fd721-0f5a-455b-b0d8-13a3856a6d70", "db_session_id": "JIVCCT1KBAG5RYITMTFS", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-02-10T05:49:30.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544170526955, "job": 1, "event": "recovery_finished"} 2024-02-10T05:49:30.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-02-10T05:49:30.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T05:49:30.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556e26d28000 2024-02-10T05:49:30.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: DB pointer 0x556e26d12000 2024-02-10T05:49:30.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-10T05:49:30.698 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-10T05:49:30.698 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** DB Stats ** 2024-02-10T05:49:30.698 INFO:journalctl@ceph.mon.a.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:49:30.698 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T05:49:30.699 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:49:30.699 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:49:30.699 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T05:49:30.699 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:49:30.699 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:49:30.699 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-02-10T05:49:30.699 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** Compaction Stats [default] ** 2024-02-10T05:49:30.699 INFO:journalctl@ceph.mon.a.smithi064.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T05:49:30.699 INFO:journalctl@ceph.mon.a.smithi064.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-10T05:49:30.699 INFO:journalctl@ceph.mon.a.smithi064.stdout: L0 2/0 81.18 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 104.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-10T05:49:30.699 INFO:journalctl@ceph.mon.a.smithi064.stdout: Sum 2/0 81.18 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 104.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-10T05:49:30.699 INFO:journalctl@ceph.mon.a.smithi064.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 104.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-10T05:49:30.699 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-02-10T05:49:30.700 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** Compaction Stats [default] ** 2024-02-10T05:49:30.700 INFO:journalctl@ceph.mon.a.smithi064.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T05:49:30.700 INFO:journalctl@ceph.mon.a.smithi064.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:49:30.700 INFO:journalctl@ceph.mon.a.smithi064.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 104.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-10T05:49:30.700 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-02-10T05:49:30.700 INFO:journalctl@ceph.mon.a.smithi064.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-10T05:49:30.700 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-02-10T05:49:30.700 INFO:journalctl@ceph.mon.a.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:49:30.700 INFO:journalctl@ceph.mon.a.smithi064.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T05:49:30.700 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T05:49:30.700 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T05:49:30.700 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T05:49:30.700 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T05:49:30.700 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative compaction: 0.00 GB write, 19.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:49:30.701 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval compaction: 0.00 GB write, 19.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:49:30.701 INFO:journalctl@ceph.mon.a.smithi064.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T05:49:30.701 INFO:journalctl@ceph.mon.a.smithi064.stdout: Block cache BinnedLRUCache@0x556e26c37090#2 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-02-10T05:49:30.701 INFO:journalctl@ceph.mon.a.smithi064.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-02-10T05:49:30.701 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-02-10T05:49:30.701 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T05:49:30.701 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: starting mon.a rank 0 at public addrs [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] at bind addrs [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:49:30.701 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: mon.a@-1(???) e1 preinit fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:49:30.701 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: mon.a@-1(???).mds e1 new map 2024-02-10T05:49:30.701 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: mon.a@-1(???).mds e1 print_map 2024-02-10T05:49:30.701 INFO:journalctl@ceph.mon.a.smithi064.stdout: e1 2024-02-10T05:49:30.701 INFO:journalctl@ceph.mon.a.smithi064.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T05:49:30.701 INFO:journalctl@ceph.mon.a.smithi064.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-10T05:49:30.701 INFO:journalctl@ceph.mon.a.smithi064.stdout: legacy client fscid: -1 2024-02-10T05:49:30.702 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2024-02-10T05:49:30.702 INFO:journalctl@ceph.mon.a.smithi064.stdout: No filesystems configured 2024-02-10T05:49:30.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T05:49:30.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:49:30.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:49:30.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:49:30.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-10T05:49:30.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 bash[25166]: 22a0019eee341f33a7f0065792d0be56b53f5fae6d028ec29d471d635912da54 2024-02-10T05:49:30.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 systemd[1]: Started Ceph mon.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:49:30.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-10T05:49:30.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: monmap e1: 1 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:49:30.702 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: fsmap 2024-02-10T05:49:30.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: osdmap e1: 0 total, 0 up, 0 in 2024-02-10T05:49:30.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:30 smithi064 ceph-mon[25185]: mgrmap e1: no daemons active 2024-02-10T05:49:32.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:32 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/2974913227' entity='client.admin' 2024-02-10T05:49:34.262 INFO:teuthology.orchestra.run.smithi064.stdout:Wrote config to /etc/ceph/ceph.conf 2024-02-10T05:49:34.453 INFO:teuthology.orchestra.run.smithi064.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:49:34.453 INFO:teuthology.orchestra.run.smithi064.stdout:Creating mgr... 2024-02-10T05:49:34.454 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying port 0.0.0.0:9283 ... 2024-02-10T05:49:34.455 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying port 0.0.0.0:8765 ... 2024-02-10T05:49:35.461 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@mgr.y 2024-02-10T05:49:35.461 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Failed to reset failed state of unit ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@mgr.y.service: Unit ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@mgr.y.service not loaded. 2024-02-10T05:49:35.728 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715.target.wants/ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@.service. 2024-02-10T05:49:36.126 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:35 smithi064 podman[25406]: 2024-02-10 05:49:35.868610317 +0000 UTC m=+0.017406333 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:49:37.376 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:37 smithi064 podman[25406]: 2024-02-10 05:49:37.001863374 +0000 UTC m=+1.150659418 container create 0209d0317adfbf50ed7401e8858d1298291832ddb496647360c139528f9e3bf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=main-290212a, org.label-schema.build-date=20240131, ceph=True, io.buildah.version=1.33.2) 2024-02-10T05:49:39.126 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:38 smithi064 podman[25406]: 2024-02-10 05:49:38.68712826 +0000 UTC m=+2.835924274 container init 0209d0317adfbf50ed7401e8858d1298291832ddb496647360c139528f9e3bf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-02-10T05:49:39.126 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:38 smithi064 podman[25406]: 2024-02-10 05:49:38.690465995 +0000 UTC m=+2.839262000 container start 0209d0317adfbf50ed7401e8858d1298291832ddb496647360c139528f9e3bf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2) 2024-02-10T05:49:39.532 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:39 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:39.255+0000 7fad46798200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:49:39.532 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:39 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:39.349+0000 7fad46798200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:49:39.546 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-10T05:49:39.546 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout disabled 2024-02-10T05:49:39.554 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-10T05:49:39.554 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-02-10T05:49:39.554 INFO:teuthology.orchestra.run.smithi064.stdout:firewalld.service is not enabled 2024-02-10T05:49:39.554 INFO:teuthology.orchestra.run.smithi064.stdout:Not possible to enable service . firewalld.service is not available 2024-02-10T05:49:39.562 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-10T05:49:39.562 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout disabled 2024-02-10T05:49:39.570 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-10T05:49:39.570 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-02-10T05:49:39.570 INFO:teuthology.orchestra.run.smithi064.stdout:firewalld.service is not enabled 2024-02-10T05:49:39.570 INFO:teuthology.orchestra.run.smithi064.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-02-10T05:49:39.570 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mgr to start... 2024-02-10T05:49:39.570 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mgr... 2024-02-10T05:49:39.876 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:39 smithi064 bash[25406]: 0209d0317adfbf50ed7401e8858d1298291832ddb496647360c139528f9e3bf3 2024-02-10T05:49:39.876 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:39 smithi064 systemd[1]: Started Ceph mgr.y for 00e86dbe-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:49:40.357 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:40 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:40.022+0000 7fad46798200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:49:40.357 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:40 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:40.116+0000 7fad46798200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:49:40.357 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:40 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:40.258+0000 7fad46798200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:49:40.357 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:40 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:40.356+0000 7fad46798200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:49:40.626 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:40 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:40.441+0000 7fad46798200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:49:40.939 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:40 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:40.711+0000 7fad46798200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T05:49:40.939 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:40 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:40.826+0000 7fad46798200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:49:40.939 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:40 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:40.916+0000 7fad46798200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:49:41.199 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:41 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:41.020+0000 7fad46798200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:49:41.199 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:41 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:41.198+0000 7fad46798200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:49:41.481 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-02-10T05:49:41.481 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-02-10T05:49:41.481 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsid": "00e86dbe-c7d8-11ee-95b8-87774f69a715", 2024-02-10T05:49:41.481 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "health": { 2024-02-10T05:49:41.481 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-10T05:49:41.482 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-10T05:49:41.482 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-10T05:49:41.482 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:49:41.482 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-10T05:49:41.482 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-10T05:49:41.482 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 0 2024-02-10T05:49:41.482 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:49:41.482 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-10T05:49:41.482 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "a" 2024-02-10T05:49:41.482 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:49:41.482 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-02-10T05:49:41.482 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-10T05:49:41.482 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:49:41.483 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-10T05:49:41.483 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-10T05:49:41.483 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:49:41.483 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-10T05:49:41.483 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:49:41.483 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-10T05:49:41.483 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-10T05:49:41.483 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-10T05:49:41.483 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-10T05:49:41.483 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-10T05:49:41.483 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-10T05:49:41.487 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:49:41.487 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-10T05:49:41.487 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-10T05:49:41.487 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-10T05:49:41.487 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-10T05:49:41.487 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-10T05:49:41.487 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-10T05:49:41.487 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-10T05:49:41.487 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-10T05:49:41.487 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-10T05:49:41.487 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:49:41.487 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-10T05:49:41.488 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:49:41.488 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-10T05:49:41.488 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-10T05:49:41.488 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:49:41.488 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-10T05:49:41.488 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-10T05:49:41.488 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-10T05:49:41.488 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-10T05:49:41.488 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-10T05:49:41.488 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-10T05:49:41.488 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "restful" 2024-02-10T05:49:41.488 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:49:41.488 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T05:49:41.488 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:49:41.489 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-10T05:49:41.489 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:49:41.489 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modified": "2024-02-10T05:49:26.002576+0000", 2024-02-10T05:49:41.489 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T05:49:41.489 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:49:41.489 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-10T05:49:41.489 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-02-10T05:49:41.489 INFO:teuthology.orchestra.run.smithi064.stdout:mgr not available, waiting (1/15)... 2024-02-10T05:49:41.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:41 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/1330729330' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T05:49:42.626 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:42 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:42.195+0000 7fad46798200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:49:42.626 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:42 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:42.330+0000 7fad46798200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:49:42.961 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:42 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:42.667+0000 7fad46798200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:49:42.962 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:42 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:42.760+0000 7fad46798200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:49:42.962 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:42 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:42.961+0000 7fad46798200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:49:43.376 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:43 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:43.149+0000 7fad46798200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:49:43.704 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:43 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:43.439+0000 7fad46798200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:49:43.704 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:43 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:43.703+0000 7fad46798200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:49:44.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:43 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/776465725' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T05:49:44.127 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:43 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:43.953+0000 7fad46798200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:49:44.432 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-02-10T05:49:44.432 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-02-10T05:49:44.432 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsid": "00e86dbe-c7d8-11ee-95b8-87774f69a715", 2024-02-10T05:49:44.432 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "health": { 2024-02-10T05:49:44.432 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-10T05:49:44.432 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-10T05:49:44.433 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-10T05:49:44.433 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:49:44.433 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-10T05:49:44.433 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-10T05:49:44.433 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 0 2024-02-10T05:49:44.433 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:49:44.433 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-10T05:49:44.433 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "a" 2024-02-10T05:49:44.433 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:49:44.433 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-02-10T05:49:44.433 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-10T05:49:44.433 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:49:44.433 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-10T05:49:44.433 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-10T05:49:44.434 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:49:44.434 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-10T05:49:44.434 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:49:44.434 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-10T05:49:44.434 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-10T05:49:44.434 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-10T05:49:44.434 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-10T05:49:44.434 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-10T05:49:44.434 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-10T05:49:44.434 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:49:44.434 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-10T05:49:44.434 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-10T05:49:44.434 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-10T05:49:44.434 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-10T05:49:44.434 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-10T05:49:44.435 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-10T05:49:44.435 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-10T05:49:44.435 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-10T05:49:44.435 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-10T05:49:44.435 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:49:44.435 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-10T05:49:44.435 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:49:44.435 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-10T05:49:44.435 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-10T05:49:44.435 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:49:44.435 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-10T05:49:44.435 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-10T05:49:44.435 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-10T05:49:44.435 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-10T05:49:44.436 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-10T05:49:44.436 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-10T05:49:44.436 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "restful" 2024-02-10T05:49:44.436 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:49:44.436 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T05:49:44.436 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:49:44.436 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-10T05:49:44.436 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:49:44.436 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modified": "2024-02-10T05:49:26.002576+0000", 2024-02-10T05:49:44.436 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T05:49:44.436 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:49:44.436 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-10T05:49:44.436 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-02-10T05:49:44.437 INFO:teuthology.orchestra.run.smithi064.stdout:mgr not available, waiting (2/15)... 2024-02-10T05:49:45.126 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:44.783+0000 7fad46798200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:49:45.626 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:45 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:45.362+0000 7fad46798200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:49:45.626 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:45 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:45.463+0000 7fad46798200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:49:46.126 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:45 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:45.682+0000 7fad46798200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:49:46.810 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:46 smithi064 ceph-mon[25185]: Activating manager daemon y 2024-02-10T05:49:46.810 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:46 smithi064 ceph-mon[25185]: mgrmap e2: y(active, starting, since 0.00293687s) 2024-02-10T05:49:46.810 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:46 smithi064 ceph-mon[25185]: from='mgr.14100 172.21.15.64:0/3302388915' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:49:46.811 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:46 smithi064 ceph-mon[25185]: from='mgr.14100 172.21.15.64:0/3302388915' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:49:46.811 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:46 smithi064 ceph-mon[25185]: from='mgr.14100 172.21.15.64:0/3302388915' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:49:46.811 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:46 smithi064 ceph-mon[25185]: from='mgr.14100 172.21.15.64:0/3302388915' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:49:46.811 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:46 smithi064 ceph-mon[25185]: from='mgr.14100 172.21.15.64:0/3302388915' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T05:49:46.811 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:46 smithi064 ceph-mon[25185]: Manager daemon y is now available 2024-02-10T05:49:46.811 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:46 smithi064 ceph-mon[25185]: from='mgr.14100 172.21.15.64:0/3302388915' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:49:46.812 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:46 smithi064 ceph-mon[25185]: from='mgr.14100 172.21.15.64:0/3302388915' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:49:46.812 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:46 smithi064 ceph-mon[25185]: from='mgr.14100 172.21.15.64:0/3302388915' entity='mgr.y' 2024-02-10T05:49:46.812 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:46 smithi064 ceph-mon[25185]: from='mgr.14100 172.21.15.64:0/3302388915' entity='mgr.y' 2024-02-10T05:49:46.812 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:46 smithi064 ceph-mon[25185]: from='mgr.14100 172.21.15.64:0/3302388915' entity='mgr.y' 2024-02-10T05:49:47.710 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-02-10T05:49:47.711 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-02-10T05:49:47.711 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsid": "00e86dbe-c7d8-11ee-95b8-87774f69a715", 2024-02-10T05:49:47.711 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "health": { 2024-02-10T05:49:47.711 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-10T05:49:47.711 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-10T05:49:47.711 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-10T05:49:47.711 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:49:47.711 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-10T05:49:47.711 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-10T05:49:47.711 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 0 2024-02-10T05:49:47.712 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:49:47.712 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-10T05:49:47.712 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "a" 2024-02-10T05:49:47.712 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:49:47.712 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_age": 16, 2024-02-10T05:49:47.712 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-10T05:49:47.712 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:49:47.712 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-10T05:49:47.712 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-10T05:49:47.712 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:49:47.712 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-10T05:49:47.712 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:49:47.712 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-10T05:49:47.713 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-10T05:49:47.713 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-10T05:49:47.713 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-10T05:49:47.713 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-10T05:49:47.713 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-10T05:49:47.716 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:49:47.717 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-10T05:49:47.717 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-10T05:49:47.717 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-10T05:49:47.717 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-10T05:49:47.717 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-10T05:49:47.717 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-10T05:49:47.717 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-10T05:49:47.717 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-10T05:49:47.717 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-10T05:49:47.717 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:49:47.717 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-10T05:49:47.717 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:49:47.717 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-10T05:49:47.717 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-10T05:49:47.718 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:49:47.718 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-10T05:49:47.718 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-10T05:49:47.718 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-10T05:49:47.718 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-10T05:49:47.718 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-10T05:49:47.718 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-10T05:49:47.718 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "restful" 2024-02-10T05:49:47.718 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:49:47.718 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T05:49:47.718 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:49:47.718 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-10T05:49:47.718 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:49:47.718 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modified": "2024-02-10T05:49:26.002576+0000", 2024-02-10T05:49:47.718 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T05:49:47.719 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:49:47.719 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-10T05:49:47.719 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-02-10T05:49:47.719 INFO:teuthology.orchestra.run.smithi064.stdout:mgr is available 2024-02-10T05:49:47.999 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:47 smithi064 ceph-mon[25185]: mgrmap e3: y(active, since 1.00936s) 2024-02-10T05:49:47.999 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:47 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/1694228703' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T05:49:48.723 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-02-10T05:49:48.723 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [global] 2024-02-10T05:49:48.723 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout fsid = 00e86dbe-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:49:48.723 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-10T05:49:48.724 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-10T05:49:48.724 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-10T05:49:48.724 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-10T05:49:48.724 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-02-10T05:49:48.724 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-10T05:49:48.724 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-10T05:49:48.724 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2024-02-10T05:49:48.724 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [osd] 2024-02-10T05:49:48.724 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-10T05:49:48.724 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-10T05:49:48.724 INFO:teuthology.orchestra.run.smithi064.stdout:Enabling cephadm module... 2024-02-10T05:49:49.013 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:48 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/1756838240' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-10T05:49:49.014 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:48 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/1756838240' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-10T05:49:49.996 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:49 smithi064 ceph-mon[25185]: mgrmap e4: y(active, since 3s) 2024-02-10T05:49:49.997 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:49 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/749240378' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-10T05:49:49.997 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:49 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: ignoring --setuser ceph since I am not root 2024-02-10T05:49:49.997 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:49 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: ignoring --setgroup ceph since I am not root 2024-02-10T05:49:50.305 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:50 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:50.304+0000 7faaa5cbe200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:49:50.559 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:50 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:50.405+0000 7faaa5cbe200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:49:50.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:50 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/749240378' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-10T05:49:50.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:50 smithi064 ceph-mon[25185]: mgrmap e5: y(active, since 4s) 2024-02-10T05:49:51.298 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:51 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:51.079+0000 7faaa5cbe200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:49:51.298 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:51 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:51.174+0000 7faaa5cbe200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:49:51.598 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:51 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:51.318+0000 7faaa5cbe200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:49:51.599 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:51 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:51.415+0000 7faaa5cbe200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:49:51.599 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:51 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:51.500+0000 7faaa5cbe200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:49:51.601 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-02-10T05:49:51.601 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-02-10T05:49:51.601 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-10T05:49:51.601 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-02-10T05:49:51.601 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-10T05:49:51.601 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-02-10T05:49:51.601 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for the mgr to restart... 2024-02-10T05:49:51.602 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mgr epoch 5... 2024-02-10T05:49:51.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:51 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/2063163491' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T05:49:51.876 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:51 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:51.771+0000 7faaa5cbe200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T05:49:52.250 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:51 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:51.885+0000 7faaa5cbe200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:49:52.251 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:51 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:51.975+0000 7faaa5cbe200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:49:52.251 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:52 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:52.073+0000 7faaa5cbe200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:49:52.626 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:52 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:52.250+0000 7faaa5cbe200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:49:53.626 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:53 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:53.231+0000 7faaa5cbe200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:49:53.626 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:53 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:53.364+0000 7faaa5cbe200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:49:53.988 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:53 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:53.698+0000 7faaa5cbe200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:49:53.988 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:53 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:53.789+0000 7faaa5cbe200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:49:53.988 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:53 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:53.987+0000 7faaa5cbe200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:49:54.376 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:54 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:54.173+0000 7faaa5cbe200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:49:54.719 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:54 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:54.459+0000 7faaa5cbe200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:49:54.720 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:54 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:54.719+0000 7faaa5cbe200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:49:55.126 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:54 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:54.957+0000 7faaa5cbe200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:49:56.126 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:55 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:55.771+0000 7faaa5cbe200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:49:56.626 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:56 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:56.349+0000 7faaa5cbe200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:49:56.626 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:56 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:56.448+0000 7faaa5cbe200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:49:57.126 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:49:56 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:49:56.666+0000 7faaa5cbe200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:49:57.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:56 smithi064 ceph-mon[25185]: Active manager daemon y restarted 2024-02-10T05:49:57.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:56 smithi064 ceph-mon[25185]: Activating manager daemon y 2024-02-10T05:49:57.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:56 smithi064 ceph-mon[25185]: osdmap e2: 0 total, 0 up, 0 in 2024-02-10T05:49:57.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:56 smithi064 ceph-mon[25185]: mgrmap e6: y(active, starting, since 0.00361209s) 2024-02-10T05:49:57.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:56 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:49:57.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:56 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T05:49:57.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:56 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:49:57.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:56 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:49:57.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:56 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:49:57.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:56 smithi064 ceph-mon[25185]: Manager daemon y is now available 2024-02-10T05:49:57.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:56 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' 2024-02-10T05:49:57.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:56 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' 2024-02-10T05:49:57.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:56 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:49:57.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:56 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:49:57.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:57 smithi064 ceph-mon[25185]: Found migration_current of "None". Setting to last migration. 2024-02-10T05:49:57.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:57 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:49:57.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:57 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:49:57.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:57 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:49:57.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:57 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' 2024-02-10T05:49:57.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:57 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' 2024-02-10T05:49:57.988 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:57 smithi064 ceph-mon[25185]: mgrmap e7: y(active, since 1.00494s) 2024-02-10T05:49:58.266 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-02-10T05:49:58.266 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-02-10T05:49:58.266 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-10T05:49:58.266 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-02-10T05:49:58.266 INFO:teuthology.orchestra.run.smithi064.stdout:mgr epoch 5 is available 2024-02-10T05:49:58.266 INFO:teuthology.orchestra.run.smithi064.stdout:Setting orchestrator backend to cephadm... 2024-02-10T05:49:58.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:58 smithi064 ceph-mon[25185]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T05:49:58.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:58 smithi064 ceph-mon[25185]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T05:49:58.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:58 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:49:59.822 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:59 smithi064 ceph-mon[25185]: [10/Feb/2024:05:49:58] ENGINE Bus STARTING 2024-02-10T05:49:59.822 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:59 smithi064 ceph-mon[25185]: [10/Feb/2024:05:49:58] ENGINE Serving on https://172.21.15.64:7150 2024-02-10T05:49:59.822 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:59 smithi064 ceph-mon[25185]: [10/Feb/2024:05:49:58] ENGINE Serving on http://172.21.15.64:8765 2024-02-10T05:49:59.822 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:59 smithi064 ceph-mon[25185]: [10/Feb/2024:05:49:58] ENGINE Bus STARTED 2024-02-10T05:49:59.822 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:59 smithi064 ceph-mon[25185]: mgrmap e8: y(active, since 2s) 2024-02-10T05:49:59.822 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:59 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' 2024-02-10T05:49:59.822 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:49:59 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:50:00.645 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout value unchanged 2024-02-10T05:50:00.645 INFO:teuthology.orchestra.run.smithi064.stdout:Generating ssh key... 2024-02-10T05:50:00.936 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:00 smithi064 ceph-mon[25185]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:50:02.070 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:01 smithi064 ceph-mon[25185]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:50:02.367 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:02 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: Generating public/private rsa key pair. 2024-02-10T05:50:02.367 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:02 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: Your identification has been saved in /tmp/tmp9xebqipe/key. 2024-02-10T05:50:02.367 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:02 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: Your public key has been saved in /tmp/tmp9xebqipe/key.pub. 2024-02-10T05:50:02.367 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:02 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: The key fingerprint is: 2024-02-10T05:50:02.367 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:02 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: SHA256:lUZfyNolG0YgcgbvM88HeDiXF82/+osjobV6mZPu9x8 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:50:02.367 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:02 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: The key's randomart image is: 2024-02-10T05:50:02.367 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:02 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: +---[RSA 3072]----+ 2024-02-10T05:50:02.367 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:02 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: | o.+ o+... | 2024-02-10T05:50:02.368 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:02 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: | = o o*+. | 2024-02-10T05:50:02.368 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:02 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: | . ++o=o | 2024-02-10T05:50:02.368 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:02 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: | . =..o. . | 2024-02-10T05:50:02.368 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:02 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: | S = . .| 2024-02-10T05:50:02.368 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:02 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: | O = .| 2024-02-10T05:50:02.368 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:02 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: | = B E | 2024-02-10T05:50:02.368 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:02 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: | . X oo .| 2024-02-10T05:50:02.368 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:02 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: | .=o+o+++| 2024-02-10T05:50:02.368 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:02 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: +----[SHA256]-----+ 2024-02-10T05:50:02.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:02 smithi064 ceph-mon[25185]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:50:02.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:02 smithi064 ceph-mon[25185]: Generating ssh key... 2024-02-10T05:50:02.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:02 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' 2024-02-10T05:50:02.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:02 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' 2024-02-10T05:50:03.809 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDd/pJ/X7hYyKrcPgFQzc/HFOF3y7BVCjTtFO7EKxNMyecdBa6Q3K0cvn17iC1sH3aOronanACbUgPcIJl+uXhCuVz3mI1bSWeJiXNMJWcE++A9Byp1XITv3O/Q1HTvDZWh5zWt2+lv4BavhGrHRem3Laa3tJbTexdcAnLl8q2jdg9N1r4RvclQB9ks/oCkZfyiKNVvHFkccYqY538oLS1FUSITl8CpLW6p+AKNeX9O2TQTJL3lkTCFzYw72xAV4nZOwfcdi0k0ApKnrK3ZV3TO/LSg6NQtkc3Ah05ekiOV+ljndW/xAG2xLIDXpeIK7OQTQIZmFvuodD72SohHPmqrd9GxJJVfwes6Q4dgUHmEcmhd7aZTRtWdmjCJaq9Kxi1onAS6XN3suDUqe6+gBkeCsJJC1I6wzeJUadp59EeyoRUyjZ47+5Al9q3km9L1JsyIzzmEMtLV8Khbqlgvr/FevD8NkzXK5bjarG42zel1/nTIxfTVl6ijI3E4wOEgyN0= ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:50:03.809 INFO:teuthology.orchestra.run.smithi064.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-10T05:50:03.809 INFO:teuthology.orchestra.run.smithi064.stdout:Adding key to root@localhost authorized_keys... 2024-02-10T05:50:03.810 INFO:teuthology.orchestra.run.smithi064.stdout:Adding host smithi064... 2024-02-10T05:50:04.092 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:03 smithi064 ceph-mon[25185]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:50:05.089 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:04 smithi064 ceph-mon[25185]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi064", "addr": "172.21.15.64", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:50:06.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:06 smithi064 ceph-mon[25185]: Deploying cephadm binary to smithi064 2024-02-10T05:50:09.341 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout Added host 'smithi064' with addr '172.21.15.64' 2024-02-10T05:50:09.342 INFO:teuthology.orchestra.run.smithi064.stdout:Deploying unmanaged mon service... 2024-02-10T05:50:09.939 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:09 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' 2024-02-10T05:50:09.940 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:09 smithi064 ceph-mon[25185]: Added host smithi064 2024-02-10T05:50:09.940 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:09 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:50:10.643 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-02-10T05:50:10.643 INFO:teuthology.orchestra.run.smithi064.stdout:Deploying unmanaged mgr service... 2024-02-10T05:50:11.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:11 smithi064 ceph-mon[25185]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:50:11.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:11 smithi064 ceph-mon[25185]: Saving service mon spec with placement count:5 2024-02-10T05:50:11.255 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:11 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' 2024-02-10T05:50:12.587 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-02-10T05:50:12.871 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:12 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' 2024-02-10T05:50:12.871 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:12 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' 2024-02-10T05:50:12.871 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:12 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' 2024-02-10T05:50:13.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:13 smithi064 ceph-mon[25185]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:50:13.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:13 smithi064 ceph-mon[25185]: Saving service mgr spec with placement count:2 2024-02-10T05:50:13.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:13 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/3591116977' entity='client.admin' 2024-02-10T05:50:14.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:14 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' 2024-02-10T05:50:15.352 INFO:teuthology.orchestra.run.smithi064.stdout:Enabling the dashboard module... 2024-02-10T05:50:15.962 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:15 smithi064 ceph-mon[25185]: from='mgr.14118 172.21.15.64:0/2806017015' entity='mgr.y' 2024-02-10T05:50:15.962 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:15 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/765643470' entity='client.admin' 2024-02-10T05:50:16.830 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:16 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/984743833' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-10T05:50:16.830 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:16 smithi064 ceph-mon[25185]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:50:17.126 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:16 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: ignoring --setuser ceph since I am not root 2024-02-10T05:50:17.126 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:16 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: ignoring --setgroup ceph since I am not root 2024-02-10T05:50:17.690 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:17 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:17.379+0000 7f7f52d92200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:50:17.690 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:17 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:17.479+0000 7f7f52d92200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:50:18.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:17 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/984743833' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-10T05:50:18.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:17 smithi064 ceph-mon[25185]: mgrmap e9: y(active, since 20s) 2024-02-10T05:50:18.427 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:18 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:18.223+0000 7f7f52d92200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:50:18.427 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:18 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:18.321+0000 7f7f52d92200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:50:18.761 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:18 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:18.474+0000 7f7f52d92200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:50:18.761 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:18 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:18.576+0000 7f7f52d92200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:50:18.761 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:18 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:18.668+0000 7f7f52d92200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:50:18.763 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-02-10T05:50:18.764 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-02-10T05:50:18.764 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-10T05:50:18.764 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-02-10T05:50:18.764 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-10T05:50:18.764 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-02-10T05:50:18.764 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for the mgr to restart... 2024-02-10T05:50:18.764 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mgr epoch 9... 2024-02-10T05:50:19.055 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:18 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/3088143155' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T05:50:19.055 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:18 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:18.950+0000 7f7f52d92200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T05:50:19.376 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:19 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:19.067+0000 7f7f52d92200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:50:19.376 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:19 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:19.159+0000 7f7f52d92200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:50:19.376 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:19 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:19.258+0000 7f7f52d92200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:50:19.876 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:19 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:19.437+0000 7f7f52d92200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:50:20.876 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:20 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:20.426+0000 7f7f52d92200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:50:20.876 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:20 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:20.560+0000 7f7f52d92200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:50:21.189 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:20 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:20.897+0000 7f7f52d92200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:50:21.189 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:20 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:20.988+0000 7f7f52d92200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:50:21.189 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:21 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:21.188+0000 7f7f52d92200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:50:21.626 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:21 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:21.376+0000 7f7f52d92200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:50:21.927 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:21 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:21.664+0000 7f7f52d92200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:50:21.928 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:21 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:21.926+0000 7f7f52d92200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:50:22.376 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:22 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:22.167+0000 7f7f52d92200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:50:23.376 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:22 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:22.990+0000 7f7f52d92200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:50:23.876 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:23 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:23.572+0000 7f7f52d92200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:50:23.876 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:23 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:23.672+0000 7f7f52d92200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:50:24.376 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:50:23 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:50:23.892+0000 7f7f52d92200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:50:24.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:23 smithi064 ceph-mon[25185]: Active manager daemon y restarted 2024-02-10T05:50:24.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:23 smithi064 ceph-mon[25185]: Activating manager daemon y 2024-02-10T05:50:24.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:23 smithi064 ceph-mon[25185]: osdmap e3: 0 total, 0 up, 0 in 2024-02-10T05:50:24.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:23 smithi064 ceph-mon[25185]: mgrmap e10: y(active, starting, since 0.00348231s) 2024-02-10T05:50:24.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:23 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:50:24.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:23 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T05:50:24.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:23 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:50:24.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:23 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:50:24.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:23 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:50:24.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:23 smithi064 ceph-mon[25185]: Manager daemon y is now available 2024-02-10T05:50:24.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:23 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:50:24.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:23 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:50:25.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:24 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:50:25.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:24 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:50:25.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:24 smithi064 ceph-mon[25185]: mgrmap e11: y(active, since 1.00613s) 2024-02-10T05:50:25.764 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2024-02-10T05:50:25.764 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-02-10T05:50:25.764 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-10T05:50:25.764 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2024-02-10T05:50:25.764 INFO:teuthology.orchestra.run.smithi064.stdout:mgr epoch 9 is available 2024-02-10T05:50:25.764 INFO:teuthology.orchestra.run.smithi064.stdout:Generating a dashboard self-signed certificate... 2024-02-10T05:50:26.533 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:26 smithi064 ceph-mon[25185]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T05:50:26.533 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:26 smithi064 ceph-mon[25185]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T05:50:26.534 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:26 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:27.103 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-02-10T05:50:27.104 INFO:teuthology.orchestra.run.smithi064.stdout:Creating initial admin user... 2024-02-10T05:50:27.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:27 smithi064 ceph-mon[25185]: [10/Feb/2024:05:50:25] ENGINE Bus STARTING 2024-02-10T05:50:27.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:27 smithi064 ceph-mon[25185]: [10/Feb/2024:05:50:25] ENGINE Serving on http://172.21.15.64:8765 2024-02-10T05:50:27.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:27 smithi064 ceph-mon[25185]: [10/Feb/2024:05:50:26] ENGINE Serving on https://172.21.15.64:7150 2024-02-10T05:50:27.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:27 smithi064 ceph-mon[25185]: [10/Feb/2024:05:50:26] ENGINE Bus STARTED 2024-02-10T05:50:27.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:27 smithi064 ceph-mon[25185]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:50:27.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:27 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:27.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:27 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:27.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:27 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:27.757 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:27 smithi064 ceph-mon[25185]: mgrmap e12: y(active, since 2s) 2024-02-10T05:50:28.522 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$gyNIDx/2caOquGZ/aBpI9ueKna6AicVqSsu4XcuChWunt.IMoSrd.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707544227, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-10T05:50:28.569 INFO:teuthology.orchestra.run.smithi064.stdout:Fetching dashboard port number... 2024-02-10T05:50:28.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:28 smithi064 ceph-mon[25185]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:50:28.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:28 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:29.545 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:29 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/53419538' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-10T05:50:29.847 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 8443 2024-02-10T05:50:29.856 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-10T05:50:29.856 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout disabled 2024-02-10T05:50:29.864 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-10T05:50:29.864 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2024-02-10T05:50:29.864 INFO:teuthology.orchestra.run.smithi064.stdout:firewalld.service is not enabled 2024-02-10T05:50:29.864 INFO:teuthology.orchestra.run.smithi064.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-10T05:50:29.866 INFO:teuthology.orchestra.run.smithi064.stdout:Ceph Dashboard is now available at: 2024-02-10T05:50:29.866 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:50:29.866 INFO:teuthology.orchestra.run.smithi064.stdout: URL: https://smithi064.front.sepia.ceph.com:8443/ 2024-02-10T05:50:29.866 INFO:teuthology.orchestra.run.smithi064.stdout: User: admin 2024-02-10T05:50:29.866 INFO:teuthology.orchestra.run.smithi064.stdout: Password: 9y5w8w5dka 2024-02-10T05:50:29.866 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:50:29.867 INFO:teuthology.orchestra.run.smithi064.stdout:Saving cluster configuration to /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config directory 2024-02-10T05:50:29.867 INFO:teuthology.orchestra.run.smithi064.stdout:Enabling autotune for osd_memory_target 2024-02-10T05:50:31.765 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:31 smithi064 ceph-mon[25185]: mgrmap e13: y(active, since 6s) 2024-02-10T05:50:32.327 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-02-10T05:50:32.327 INFO:teuthology.orchestra.run.smithi064.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-10T05:50:32.327 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:50:32.327 INFO:teuthology.orchestra.run.smithi064.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:50:32.327 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:50:32.327 INFO:teuthology.orchestra.run.smithi064.stdout:Or, if you are only running a single cluster on this host: 2024-02-10T05:50:32.328 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:50:32.328 INFO:teuthology.orchestra.run.smithi064.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-10T05:50:32.328 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:50:32.328 INFO:teuthology.orchestra.run.smithi064.stdout:Please consider enabling telemetry to help improve Ceph: 2024-02-10T05:50:32.328 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:50:32.328 INFO:teuthology.orchestra.run.smithi064.stdout: ceph telemetry on 2024-02-10T05:50:32.328 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:50:32.328 INFO:teuthology.orchestra.run.smithi064.stdout:For more information see: 2024-02-10T05:50:32.328 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:50:32.328 INFO:teuthology.orchestra.run.smithi064.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-02-10T05:50:32.329 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:50:32.329 INFO:teuthology.orchestra.run.smithi064.stdout:Bootstrap complete. 2024-02-10T05:50:32.372 INFO:tasks.cephadm:Fetching config... 2024-02-10T05:50:32.372 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T05:50:32.372 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-10T05:50:32.388 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-10T05:50:32.388 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T05:50:32.388 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-10T05:50:32.443 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-10T05:50:32.443 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T05:50:32.443 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-02-10T05:50:32.509 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-10T05:50:32.510 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T05:50:32.510 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-10T05:50:32.566 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-10T05:50:32.566 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDd/pJ/X7hYyKrcPgFQzc/HFOF3y7BVCjTtFO7EKxNMyecdBa6Q3K0cvn17iC1sH3aOronanACbUgPcIJl+uXhCuVz3mI1bSWeJiXNMJWcE++A9Byp1XITv3O/Q1HTvDZWh5zWt2+lv4BavhGrHRem3Laa3tJbTexdcAnLl8q2jdg9N1r4RvclQB9ks/oCkZfyiKNVvHFkccYqY538oLS1FUSITl8CpLW6p+AKNeX9O2TQTJL3lkTCFzYw72xAV4nZOwfcdi0k0ApKnrK3ZV3TO/LSg6NQtkc3Ah05ekiOV+ljndW/xAG2xLIDXpeIK7OQTQIZmFvuodD72SohHPmqrd9GxJJVfwes6Q4dgUHmEcmhd7aZTRtWdmjCJaq9Kxi1onAS6XN3suDUqe6+gBkeCsJJC1I6wzeJUadp59EeyoRUyjZ47+5Al9q3km9L1JsyIzzmEMtLV8Khbqlgvr/FevD8NkzXK5bjarG42zel1/nTIxfTVl6ijI3E4wOEgyN0= ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T05:50:32.659 INFO:teuthology.orchestra.run.smithi064.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDd/pJ/X7hYyKrcPgFQzc/HFOF3y7BVCjTtFO7EKxNMyecdBa6Q3K0cvn17iC1sH3aOronanACbUgPcIJl+uXhCuVz3mI1bSWeJiXNMJWcE++A9Byp1XITv3O/Q1HTvDZWh5zWt2+lv4BavhGrHRem3Laa3tJbTexdcAnLl8q2jdg9N1r4RvclQB9ks/oCkZfyiKNVvHFkccYqY538oLS1FUSITl8CpLW6p+AKNeX9O2TQTJL3lkTCFzYw72xAV4nZOwfcdi0k0ApKnrK3ZV3TO/LSg6NQtkc3Ah05ekiOV+ljndW/xAG2xLIDXpeIK7OQTQIZmFvuodD72SohHPmqrd9GxJJVfwes6Q4dgUHmEcmhd7aZTRtWdmjCJaq9Kxi1onAS6XN3suDUqe6+gBkeCsJJC1I6wzeJUadp59EeyoRUyjZ47+5Al9q3km9L1JsyIzzmEMtLV8Khbqlgvr/FevD8NkzXK5bjarG42zel1/nTIxfTVl6ijI3E4wOEgyN0= ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:50:32.688 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDd/pJ/X7hYyKrcPgFQzc/HFOF3y7BVCjTtFO7EKxNMyecdBa6Q3K0cvn17iC1sH3aOronanACbUgPcIJl+uXhCuVz3mI1bSWeJiXNMJWcE++A9Byp1XITv3O/Q1HTvDZWh5zWt2+lv4BavhGrHRem3Laa3tJbTexdcAnLl8q2jdg9N1r4RvclQB9ks/oCkZfyiKNVvHFkccYqY538oLS1FUSITl8CpLW6p+AKNeX9O2TQTJL3lkTCFzYw72xAV4nZOwfcdi0k0ApKnrK3ZV3TO/LSg6NQtkc3Ah05ekiOV+ljndW/xAG2xLIDXpeIK7OQTQIZmFvuodD72SohHPmqrd9GxJJVfwes6Q4dgUHmEcmhd7aZTRtWdmjCJaq9Kxi1onAS6XN3suDUqe6+gBkeCsJJC1I6wzeJUadp59EeyoRUyjZ47+5Al9q3km9L1JsyIzzmEMtLV8Khbqlgvr/FevD8NkzXK5bjarG42zel1/nTIxfTVl6ijI3E4wOEgyN0= ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T05:50:32.737 INFO:teuthology.orchestra.run.smithi176.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDd/pJ/X7hYyKrcPgFQzc/HFOF3y7BVCjTtFO7EKxNMyecdBa6Q3K0cvn17iC1sH3aOronanACbUgPcIJl+uXhCuVz3mI1bSWeJiXNMJWcE++A9Byp1XITv3O/Q1HTvDZWh5zWt2+lv4BavhGrHRem3Laa3tJbTexdcAnLl8q2jdg9N1r4RvclQB9ks/oCkZfyiKNVvHFkccYqY538oLS1FUSITl8CpLW6p+AKNeX9O2TQTJL3lkTCFzYw72xAV4nZOwfcdi0k0ApKnrK3ZV3TO/LSg6NQtkc3Ah05ekiOV+ljndW/xAG2xLIDXpeIK7OQTQIZmFvuodD72SohHPmqrd9GxJJVfwes6Q4dgUHmEcmhd7aZTRtWdmjCJaq9Kxi1onAS6XN3suDUqe6+gBkeCsJJC1I6wzeJUadp59EeyoRUyjZ47+5Al9q3km9L1JsyIzzmEMtLV8Khbqlgvr/FevD8NkzXK5bjarG42zel1/nTIxfTVl6ijI3E4wOEgyN0= ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:50:32.752 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-10T05:50:32.886 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:32 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/853200021' entity='client.admin' 2024-02-10T05:50:32.994 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:50:36.041 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:35 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:36.041 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:35 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:36.041 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:35 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi064", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:50:36.041 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:35 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:36.041 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:35 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:50:36.041 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:35 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:36.041 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:35 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/3036751169' entity='client.admin' 2024-02-10T05:50:36.041 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:35 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:50:36.041 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:35 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:36.074 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-10T05:50:36.074 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-10T05:50:36.325 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:50:38.909 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi176 2024-02-10T05:50:38.910 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-10T05:50:38.910 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/etc/ceph/ceph.conf 2024-02-10T05:50:38.928 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-10T05:50:38.928 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:50:38.985 INFO:tasks.cephadm:Adding host smithi176 to orchestrator... 2024-02-10T05:50:38.986 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph orch host add smithi176 2024-02-10T05:50:39.222 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:50:39.312 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:39 smithi064 ceph-mon[25185]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:50:39.312 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:39 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:39.313 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:39 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:50:39.313 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:39 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:50:39.313 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:39 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:50:39.313 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:39 smithi064 ceph-mon[25185]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-10T05:50:39.313 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:39 smithi064 ceph-mon[25185]: Updating smithi064:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:50:39.313 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:39 smithi064 ceph-mon[25185]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:50:39.313 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:39 smithi064 ceph-mon[25185]: Updating smithi064:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:50:39.313 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:39 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:39.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:39 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:39.314 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:39 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:41.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:41 smithi064 ceph-mon[25185]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi176", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:50:42.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:42 smithi064 ceph-mon[25185]: Deploying cephadm binary to smithi176 2024-02-10T05:50:45.218 INFO:teuthology.orchestra.run.smithi064.stdout:Added host 'smithi176' with addr '172.21.15.176' 2024-02-10T05:50:45.219 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:44 smithi064 ceph-mon[25185]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:50:45.800 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph orch host ls --format=json 2024-02-10T05:50:46.042 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:50:46.476 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:46 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:46.477 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:46 smithi064 ceph-mon[25185]: Added host smithi176 2024-02-10T05:50:46.477 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:46 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:50:46.477 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:46 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:47.433 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:47 smithi064 ceph-mon[25185]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:50:47.433 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:47.751 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:50:47.751 INFO:teuthology.orchestra.run.smithi064.stdout:[{"addr": "172.21.15.64", "hostname": "smithi064", "labels": [], "status": ""}, {"addr": "172.21.15.176", "hostname": "smithi176", "labels": [], "status": ""}] 2024-02-10T05:50:48.337 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-10T05:50:48.338 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph osd crush tunables default 2024-02-10T05:50:48.576 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:50:48.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:48 smithi064 ceph-mon[25185]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:50:48.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:48 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:50.070 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:49 smithi064 ceph-mon[25185]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:50:50.821 INFO:teuthology.orchestra.run.smithi064.stderr:adjusted tunables profile to default 2024-02-10T05:50:51.106 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:50 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/1195470852' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-10T05:50:51.452 INFO:tasks.cephadm:Adding mon.a on smithi064 2024-02-10T05:50:51.453 INFO:tasks.cephadm:Adding mon.c on smithi064 2024-02-10T05:50:51.453 INFO:tasks.cephadm:Adding mon.b on smithi176 2024-02-10T05:50:51.453 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph orch apply mon '3;smithi064:172.21.15.64=a;smithi064:[v2:172.21.15.64:3301,v1:172.21.15.64:6790]=c;smithi176:172.21.15.176=b' 2024-02-10T05:50:51.687 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T05:50:51.734 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-10T05:50:52.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:51 smithi064 ceph-mon[25185]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:50:52.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:51 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/1195470852' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-10T05:50:52.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:51 smithi064 ceph-mon[25185]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:50:52.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:51 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:52.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:51 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:52.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:51 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:52.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:51 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:52.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:51 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:50:52.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:51 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:50:52.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:51 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:50:53.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:52 smithi064 ceph-mon[25185]: Updating smithi176:/etc/ceph/ceph.conf 2024-02-10T05:50:53.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:52 smithi064 ceph-mon[25185]: Updating smithi176:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:50:53.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:52 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:53.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:52 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:53.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:52 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:53.383 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled mon update... 2024-02-10T05:50:53.950 DEBUG:teuthology.orchestra.run.smithi064:mon.c> sudo journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@mon.c.service 2024-02-10T05:50:53.952 DEBUG:teuthology.orchestra.run.smithi176:mon.b> sudo journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@mon.b.service 2024-02-10T05:50:53.954 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T05:50:53.955 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:50:54.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:53 smithi064 ceph-mon[25185]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:50:54.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:53 smithi064 ceph-mon[25185]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:50:54.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:53 smithi064 ceph-mon[25185]: Updating smithi176:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:50:54.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:53 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:54.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:53 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:50:54.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:53 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:50:54.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:53 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:50:54.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:53 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:54.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:53 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:50:54.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:53 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:50:55.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:54 smithi064 ceph-mon[25185]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi064:172.21.15.64=a;smithi064:[v2:172.21.15.64:3301,v1:172.21.15.64:6790]=c;smithi176:172.21.15.176=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:50:55.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:54 smithi064 ceph-mon[25185]: Saving service mon spec with placement smithi064:172.21.15.64=a;smithi064:[v2:172.21.15.64:3301,v1:172.21.15.64:6790]=c;smithi176:172.21.15.176=b;count:3 2024-02-10T05:50:55.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:54 smithi064 ceph-mon[25185]: Deploying daemon mon.b on smithi176 2024-02-10T05:50:55.474 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:50:56.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:55 smithi064 ceph-mon[25185]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:50:58.063 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-10T05:50:58.063 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":1,"fsid":"00e86dbe-c7d8-11ee-95b8-87774f69a715","modified":"2024-02-10T05:49:23.053612Z","created":"2024-02-10T05:49:23.053612Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T05:50:58.065 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 1 2024-02-10T05:50:58.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:50:57 smithi064 ceph-mon[25185]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:50:58.169 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-10T05:50:58.169 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-10T05:50:58.169 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-10T05:50:58.170 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-10T05:50:58.170 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-10T05:50:58.170 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-10T05:50:58.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T05:50:58.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:50:58.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:50:58.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:50:58.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/1195470852' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-10T05:50:58.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:50:58.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/1195470852' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-10T05:50:58.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:50:58.490 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:58.490 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:58.490 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:58.490 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:58.490 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:50:58.490 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:50:58.490 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:50:58.490 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: Updating smithi176:/etc/ceph/ceph.conf 2024-02-10T05:50:58.490 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: Updating smithi176:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:50:58.490 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:58.490 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:58.490 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:58.490 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:50:58.491 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:50:58.491 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: Updating smithi176:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:50:58.491 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:58.491 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:50:58.491 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:50:58.491 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:50:58.491 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:50:58.491 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:50:58.491 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:50:58.491 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi064:172.21.15.64=a;smithi064:[v2:172.21.15.64:3301,v1:172.21.15.64:6790]=c;smithi176:172.21.15.176=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:50:58.491 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: Saving service mon spec with placement smithi064:172.21.15.64=a;smithi064:[v2:172.21.15.64:3301,v1:172.21.15.64:6790]=c;smithi176:172.21.15.176=b;count:3 2024-02-10T05:50:58.491 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: Deploying daemon mon.b on smithi176 2024-02-10T05:50:58.491 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:50:58.491 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:50:58.491 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 ceph-mon[30294]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-10T05:50:58.492 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 bash[30263]: 87e8c2896dc7f15f6e0639bd26a5eec0f71cdee19514dd3825cd404bb8554a9a 2024-02-10T05:50:58.492 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:50:58 smithi176 systemd[1]: Started Ceph mon.b for 00e86dbe-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:50:59.779 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T05:50:59.779 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:51:00.023 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:51:03.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:03 smithi064 ceph-mon[25185]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:03.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:03 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:51:03.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:03 smithi064 ceph-mon[25185]: mon.a calling monitor election 2024-02-10T05:51:03.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:03 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:51:03.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:03 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:51:03.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:03 smithi064 ceph-mon[25185]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:03.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:03 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:51:03.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:03 smithi064 ceph-mon[25185]: mon.b calling monitor election 2024-02-10T05:51:03.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:03 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:51:03.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:03 smithi064 ceph-mon[25185]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:03.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:03 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:51:03.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:03 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:51:03.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:03 smithi064 ceph-mon[25185]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-10T05:51:03.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:03 smithi064 ceph-mon[25185]: monmap e2: 2 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:51:03.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:03 smithi064 ceph-mon[25185]: fsmap 2024-02-10T05:51:03.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:03 smithi064 ceph-mon[25185]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:51:03.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:03 smithi064 ceph-mon[25185]: mgrmap e13: y(active, since 39s) 2024-02-10T05:51:03.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:03 smithi064 ceph-mon[25185]: overall HEALTH_OK 2024-02-10T05:51:03.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:03 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:03.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:03 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:03.738 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:03 smithi176 ceph-mon[30294]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:03.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:03 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:51:03.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:03 smithi176 ceph-mon[30294]: mon.a calling monitor election 2024-02-10T05:51:03.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:03 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:51:03.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:03 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:51:03.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:03 smithi176 ceph-mon[30294]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:03.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:03 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:51:03.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:03 smithi176 ceph-mon[30294]: mon.b calling monitor election 2024-02-10T05:51:03.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:03 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:51:03.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:03 smithi176 ceph-mon[30294]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:03.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:03 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:51:03.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:03 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:51:03.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:03 smithi176 ceph-mon[30294]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-10T05:51:03.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:03 smithi176 ceph-mon[30294]: monmap e2: 2 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:51:03.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:03 smithi176 ceph-mon[30294]: fsmap 2024-02-10T05:51:03.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:03 smithi176 ceph-mon[30294]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:51:03.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:03 smithi176 ceph-mon[30294]: mgrmap e13: y(active, since 39s) 2024-02-10T05:51:03.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:03 smithi176 ceph-mon[30294]: overall HEALTH_OK 2024-02-10T05:51:03.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:03 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:03.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:03 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:04.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:04 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:04.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:04 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:51:04.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:04 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:04.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:04 smithi064 ceph-mon[25185]: Deploying daemon mon.c on smithi064 2024-02-10T05:51:04.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:04 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:51:04.667 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:04 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:04.667 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:04 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:51:04.667 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:04 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:04.667 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:04 smithi176 ceph-mon[30294]: Deploying daemon mon.c on smithi064 2024-02-10T05:51:04.667 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:04 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:51:04.667 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-10T05:51:04.667 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":2,"fsid":"00e86dbe-c7d8-11ee-95b8-87774f69a715","modified":"2024-02-10T05:50:58.175229Z","created":"2024-02-10T05:49:23.053612Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-10T05:51:04.669 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 2 2024-02-10T05:51:05.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:05 smithi064 ceph-mon[25185]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:05.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:05 smithi064 ceph-mon[25185]: from='client.? 172.21.15.176:0/1577958389' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:51:05.738 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:05 smithi176 ceph-mon[30294]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:05.738 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:05 smithi176 ceph-mon[30294]: from='client.? 172.21.15.176:0/1577958389' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:51:06.322 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T05:51:06.322 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:51:06.567 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:51:06.700 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T05:51:06.700 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T05:51:06.700 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T05:51:06.700 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T05:51:06.700 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-10T05:51:06.700 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.max_background_jobs: 2 2024-02-10T05:51:06.701 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.max_background_compactions: -1 2024-02-10T05:51:06.701 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.max_subcompactions: 1 2024-02-10T05:51:06.701 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T05:51:06.701 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T05:51:06.701 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T05:51:06.701 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T05:51:06.701 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T05:51:06.701 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T05:51:06.701 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T05:51:06.701 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T05:51:06.701 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.max_open_files: -1 2024-02-10T05:51:06.701 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T05:51:06.701 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T05:51:06.701 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T05:51:06.702 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T05:51:06.702 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.max_background_flushes: -1 2024-02-10T05:51:06.702 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Compression algorithms supported: 2024-02-10T05:51:06.702 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T05:51:06.702 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: kZSTD supported: 0 2024-02-10T05:51:06.702 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: kXpressCompression supported: 0 2024-02-10T05:51:06.702 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T05:51:06.702 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: kZlibCompression supported: 1 2024-02-10T05:51:06.702 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: kSnappyCompression supported: 1 2024-02-10T05:51:06.702 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: kLZ4Compression supported: 1 2024-02-10T05:51:06.702 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: kBZip2Compression supported: 0 2024-02-10T05:51:06.702 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T05:51:06.702 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-10T05:51:06.702 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-02-10T05:51:06.703 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-10T05:51:06.703 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T05:51:06.703 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.merge_operator: 2024-02-10T05:51:06.703 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.compaction_filter: None 2024-02-10T05:51:06.703 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T05:51:06.703 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T05:51:06.704 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T05:51:06.704 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T05:51:06.705 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fa176e19c0) 2024-02-10T05:51:06.705 INFO:journalctl@ceph.mon.c.smithi064.stdout: cache_index_and_filter_blocks: 1 2024-02-10T05:51:06.705 INFO:journalctl@ceph.mon.c.smithi064.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T05:51:06.705 INFO:journalctl@ceph.mon.c.smithi064.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T05:51:06.705 INFO:journalctl@ceph.mon.c.smithi064.stdout: pin_top_level_index_and_filter: 1 2024-02-10T05:51:06.705 INFO:journalctl@ceph.mon.c.smithi064.stdout: index_type: 0 2024-02-10T05:51:06.705 INFO:journalctl@ceph.mon.c.smithi064.stdout: data_block_index_type: 0 2024-02-10T05:51:06.706 INFO:journalctl@ceph.mon.c.smithi064.stdout: index_shortening: 1 2024-02-10T05:51:06.706 INFO:journalctl@ceph.mon.c.smithi064.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T05:51:06.706 INFO:journalctl@ceph.mon.c.smithi064.stdout: checksum: 4 2024-02-10T05:51:06.706 INFO:journalctl@ceph.mon.c.smithi064.stdout: no_block_cache: 0 2024-02-10T05:51:06.706 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_cache: 0x55fa176e5090 2024-02-10T05:51:06.706 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_cache_name: BinnedLRUCache 2024-02-10T05:51:06.706 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_cache_options: 2024-02-10T05:51:06.706 INFO:journalctl@ceph.mon.c.smithi064.stdout: capacity : 536870912 2024-02-10T05:51:06.707 INFO:journalctl@ceph.mon.c.smithi064.stdout: num_shard_bits : 4 2024-02-10T05:51:06.707 INFO:journalctl@ceph.mon.c.smithi064.stdout: strict_capacity_limit : 0 2024-02-10T05:51:06.707 INFO:journalctl@ceph.mon.c.smithi064.stdout: high_pri_pool_ratio: 0.000 2024-02-10T05:51:06.707 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_cache_compressed: (nil) 2024-02-10T05:51:06.707 INFO:journalctl@ceph.mon.c.smithi064.stdout: persistent_cache: (nil) 2024-02-10T05:51:06.707 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_size: 4096 2024-02-10T05:51:06.707 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_size_deviation: 10 2024-02-10T05:51:06.707 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_restart_interval: 16 2024-02-10T05:51:06.708 INFO:journalctl@ceph.mon.c.smithi064.stdout: index_block_restart_interval: 1 2024-02-10T05:51:06.708 INFO:journalctl@ceph.mon.c.smithi064.stdout: metadata_block_size: 4096 2024-02-10T05:51:06.708 INFO:journalctl@ceph.mon.c.smithi064.stdout: partition_filters: 0 2024-02-10T05:51:06.708 INFO:journalctl@ceph.mon.c.smithi064.stdout: use_delta_encoding: 1 2024-02-10T05:51:06.708 INFO:journalctl@ceph.mon.c.smithi064.stdout: filter_policy: bloomfilter 2024-02-10T05:51:06.708 INFO:journalctl@ceph.mon.c.smithi064.stdout: whole_key_filtering: 1 2024-02-10T05:51:06.709 INFO:journalctl@ceph.mon.c.smithi064.stdout: verify_compression: 0 2024-02-10T05:51:06.709 INFO:journalctl@ceph.mon.c.smithi064.stdout: read_amp_bytes_per_bit: 0 2024-02-10T05:51:06.709 INFO:journalctl@ceph.mon.c.smithi064.stdout: format_version: 5 2024-02-10T05:51:06.709 INFO:journalctl@ceph.mon.c.smithi064.stdout: enable_index_compression: 1 2024-02-10T05:51:06.709 INFO:journalctl@ceph.mon.c.smithi064.stdout: block_align: 0 2024-02-10T05:51:06.709 INFO:journalctl@ceph.mon.c.smithi064.stdout: max_auto_readahead_size: 262144 2024-02-10T05:51:06.710 INFO:journalctl@ceph.mon.c.smithi064.stdout: prepopulate_block_cache: 0 2024-02-10T05:51:06.710 INFO:journalctl@ceph.mon.c.smithi064.stdout: initial_auto_readahead_size: 8192 2024-02-10T05:51:06.710 INFO:journalctl@ceph.mon.c.smithi064.stdout: num_file_reads_for_auto_readahead: 2 2024-02-10T05:51:06.710 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T05:51:06.710 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T05:51:06.710 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.compression: NoCompression 2024-02-10T05:51:06.710 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T05:51:06.711 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T05:51:06.711 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T05:51:06.711 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.num_levels: 7 2024-02-10T05:51:06.711 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T05:51:06.711 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T05:51:06.711 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T05:51:06.711 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T05:51:06.712 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T05:51:06.712 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T05:51:06.712 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T05:51:06.712 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:51:06.712 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T05:51:06.712 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T05:51:06.712 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-10T05:51:06.712 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-10T05:51:06.713 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T05:51:06.713 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T05:51:06.713 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T05:51:06.713 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T05:51:06.713 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:51:06.713 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-10T05:51:06.713 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T05:51:06.713 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T05:51:06.714 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-10T05:51:06.714 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T05:51:06.714 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T05:51:06.714 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T05:51:06.714 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T05:51:06.714 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T05:51:06.714 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T05:51:06.715 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T05:51:06.715 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T05:51:06.715 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T05:51:06.715 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T05:51:06.715 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T05:51:06.715 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T05:51:06.716 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T05:51:06.716 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T05:51:06.716 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T05:51:06.716 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T05:51:06.716 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T05:51:06.716 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-10T05:51:06.717 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.arena_block_size: 1048576 2024-02-10T05:51:06.717 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T05:51:06.717 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T05:51:06.717 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T05:51:06.717 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T05:51:06.717 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T05:51:06.717 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T05:51:06.717 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T05:51:06.717 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T05:51:06.717 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T05:51:06.717 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T05:51:06.717 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T05:51:06.717 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T05:51:06.718 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T05:51:06.718 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.table_properties_collectors: 2024-02-10T05:51:06.718 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.inplace_update_support: 0 2024-02-10T05:51:06.718 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T05:51:06.718 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T05:51:06.718 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T05:51:06.718 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T05:51:06.718 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.bloom_locality: 0 2024-02-10T05:51:06.718 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.max_successive_merges: 0 2024-02-10T05:51:06.718 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T05:51:06.718 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T05:51:06.718 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T05:51:06.718 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T05:51:06.718 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.ttl: 2592000 2024-02-10T05:51:06.719 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T05:51:06.719 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-10T05:51:06.719 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-10T05:51:06.719 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.enable_blob_files: false 2024-02-10T05:51:06.719 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.min_blob_size: 0 2024-02-10T05:51:06.719 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T05:51:06.719 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T05:51:06.719 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T05:51:06.719 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T05:51:06.719 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-10T05:51:06.719 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-10T05:51:06.719 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.blob_file_starting_level: 0 2024-02-10T05:51:06.719 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-10T05:51:06.719 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T05:51:06.720 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-02-10T05:51:06.720 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0fa75bcb-b988-4e19-a1d7-54fef41d3d7e 2024-02-10T05:51:06.720 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544266698325, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-10T05:51:06.720 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-02-10T05:51:06.720 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544266698712, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707544266, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0fa75bcb-b988-4e19-a1d7-54fef41d3d7e", "db_session_id": "8JP94WT9EC9KXLU8CMR5", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-02-10T05:51:06.720 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707544266698793, "job": 1, "event": "recovery_finished"} 2024-02-10T05:51:06.720 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-02-10T05:51:06.720 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T05:51:06.720 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fa177d6000 2024-02-10T05:51:06.720 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: DB pointer 0x55fa177c0000 2024-02-10T05:51:06.720 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-10T05:51:06.720 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-10T05:51:06.720 INFO:journalctl@ceph.mon.c.smithi064.stdout: ** DB Stats ** 2024-02-10T05:51:06.721 INFO:journalctl@ceph.mon.c.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:51:06.721 INFO:journalctl@ceph.mon.c.smithi064.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T05:51:06.721 INFO:journalctl@ceph.mon.c.smithi064.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:51:06.721 INFO:journalctl@ceph.mon.c.smithi064.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:51:06.721 INFO:journalctl@ceph.mon.c.smithi064.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T05:51:06.721 INFO:journalctl@ceph.mon.c.smithi064.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:51:06.721 INFO:journalctl@ceph.mon.c.smithi064.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:51:06.721 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-02-10T05:51:06.721 INFO:journalctl@ceph.mon.c.smithi064.stdout: ** Compaction Stats [default] ** 2024-02-10T05:51:06.721 INFO:journalctl@ceph.mon.c.smithi064.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T05:51:06.721 INFO:journalctl@ceph.mon.c.smithi064.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-10T05:51:06.721 INFO:journalctl@ceph.mon.c.smithi064.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-10T05:51:06.721 INFO:journalctl@ceph.mon.c.smithi064.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-10T05:51:06.721 INFO:journalctl@ceph.mon.c.smithi064.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-10T05:51:06.722 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-02-10T05:51:06.722 INFO:journalctl@ceph.mon.c.smithi064.stdout: ** Compaction Stats [default] ** 2024-02-10T05:51:06.722 INFO:journalctl@ceph.mon.c.smithi064.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T05:51:06.722 INFO:journalctl@ceph.mon.c.smithi064.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:51:06.722 INFO:journalctl@ceph.mon.c.smithi064.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-10T05:51:06.722 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-02-10T05:51:06.722 INFO:journalctl@ceph.mon.c.smithi064.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-10T05:51:06.722 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-02-10T05:51:06.722 INFO:journalctl@ceph.mon.c.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:51:06.722 INFO:journalctl@ceph.mon.c.smithi064.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T05:51:06.722 INFO:journalctl@ceph.mon.c.smithi064.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T05:51:06.722 INFO:journalctl@ceph.mon.c.smithi064.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T05:51:06.722 INFO:journalctl@ceph.mon.c.smithi064.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T05:51:06.722 INFO:journalctl@ceph.mon.c.smithi064.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T05:51:06.722 INFO:journalctl@ceph.mon.c.smithi064.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:51:06.723 INFO:journalctl@ceph.mon.c.smithi064.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:51:06.723 INFO:journalctl@ceph.mon.c.smithi064.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T05:51:06.723 INFO:journalctl@ceph.mon.c.smithi064.stdout: Block cache BinnedLRUCache@0x55fa176e5090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-02-10T05:51:06.723 INFO:journalctl@ceph.mon.c.smithi064.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-02-10T05:51:06.723 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-02-10T05:51:06.723 INFO:journalctl@ceph.mon.c.smithi064.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T05:51:06.723 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-02-10T05:51:06.723 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: using public_addrv [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-02-10T05:51:06.723 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: starting mon.c rank -1 at public addrs [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] at bind addrs [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:51:06.723 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: mon.c@-1(???) e0 preinit fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:51:07.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: mon.c@-1(synchronizing).mds e1 new map 2024-02-10T05:51:07.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: mon.c@-1(synchronizing).mds e1 print_map 2024-02-10T05:51:07.127 INFO:journalctl@ceph.mon.c.smithi064.stdout: e1 2024-02-10T05:51:07.127 INFO:journalctl@ceph.mon.c.smithi064.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T05:51:07.127 INFO:journalctl@ceph.mon.c.smithi064.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-10T05:51:07.127 INFO:journalctl@ceph.mon.c.smithi064.stdout: legacy client fscid: -1 2024-02-10T05:51:07.127 INFO:journalctl@ceph.mon.c.smithi064.stdout: 2024-02-10T05:51:07.127 INFO:journalctl@ceph.mon.c.smithi064.stdout: No filesystems configured 2024-02-10T05:51:07.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-10T05:51:07.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-10T05:51:07.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-10T05:51:07.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-10T05:51:07.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-10T05:51:07.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-10T05:51:07.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T05:51:07.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:51:07.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:51:07.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:51:07.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/1195470852' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-10T05:51:07.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:07.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/1195470852' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-10T05:51:07.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:51:07.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:07.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:07.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:07.129 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:07.129 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:51:07.129 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:07.129 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:51:07.129 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: Updating smithi176:/etc/ceph/ceph.conf 2024-02-10T05:51:07.129 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: Updating smithi176:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:51:07.129 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:07.129 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:07.129 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:07.129 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:51:07.129 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:07.129 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: Updating smithi176:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:51:07.129 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:07.130 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:07.130 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:07.130 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:51:07.130 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:07.130 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:51:07.130 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:07.130 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi064:172.21.15.64=a;smithi064:[v2:172.21.15.64:3301,v1:172.21.15.64:6790]=c;smithi176:172.21.15.176=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:07.130 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: Saving service mon spec with placement smithi064:172.21.15.64=a;smithi064:[v2:172.21.15.64:3301,v1:172.21.15.64:6790]=c;smithi176:172.21.15.176=b;count:3 2024-02-10T05:51:07.130 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: Deploying daemon mon.b on smithi176 2024-02-10T05:51:07.130 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:07.130 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:07.130 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:07.131 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:51:07.131 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: mon.a calling monitor election 2024-02-10T05:51:07.131 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:51:07.131 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:51:07.131 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:07.131 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:51:07.131 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: mon.b calling monitor election 2024-02-10T05:51:07.131 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:51:07.131 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:07.131 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:51:07.131 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:51:07.131 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-10T05:51:07.131 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: monmap e2: 2 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:51:07.131 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: fsmap 2024-02-10T05:51:07.131 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:51:07.132 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: mgrmap e13: y(active, since 39s) 2024-02-10T05:51:07.132 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: overall HEALTH_OK 2024-02-10T05:51:07.132 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:07.132 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:07.132 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:07.132 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:51:07.132 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:07.132 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: Deploying daemon mon.c on smithi064 2024-02-10T05:51:07.132 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:51:07.132 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:07.132 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: from='client.? 172.21.15.176:0/1577958389' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:51:07.132 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 ceph-mon[38958]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-10T05:51:07.132 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 bash[38930]: de9d442387609cab7fa63eef16c797782ca95d8be20845a3dfc6c14573efb88f 2024-02-10T05:51:07.133 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:06 smithi064 systemd[1]: Started Ceph mon.c for 00e86dbe-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:51:12.104 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:12.104 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:51:12.104 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: mon.a calling monitor election 2024-02-10T05:51:12.105 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: mon.b calling monitor election 2024-02-10T05:51:12.105 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:51:12.105 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:51:12.105 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:51:12.105 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:12.105 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:51:12.105 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:51:12.105 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:12.105 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:51:12.105 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:51:12.105 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-10T05:51:12.106 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: monmap e3: 3 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0],c=[v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:51:12.106 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: fsmap 2024-02-10T05:51:12.106 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:51:12.106 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: mgrmap e13: y(active, since 47s) 2024-02-10T05:51:12.106 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-02-10T05:51:12.106 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-02-10T05:51:12.106 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-02-10T05:51:12.106 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: mon.c (rank 2) addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] is down (out of quorum) 2024-02-10T05:51:12.106 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:12.106 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:12.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:12.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:11 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:12.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:12.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:51:12.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: mon.a calling monitor election 2024-02-10T05:51:12.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: mon.b calling monitor election 2024-02-10T05:51:12.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:51:12.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:51:12.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:51:12.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:12.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:51:12.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:51:12.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:12.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:51:12.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:51:12.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-10T05:51:12.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: monmap e3: 3 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0],c=[v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:51:12.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: fsmap 2024-02-10T05:51:12.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:51:12.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: mgrmap e13: y(active, since 47s) 2024-02-10T05:51:12.128 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-02-10T05:51:12.128 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-02-10T05:51:12.128 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-02-10T05:51:12.128 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: mon.c (rank 2) addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] is down (out of quorum) 2024-02-10T05:51:12.128 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:12.128 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:12.128 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:12.128 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:11 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:12.689 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-02-10T05:51:12.690 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":3,"fsid":"00e86dbe-c7d8-11ee-95b8-87774f69a715","modified":"2024-02-10T05:51:06.722618Z","created":"2024-02-10T05:49:23.053612Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3301","nonce":0},{"type":"v1","addr":"172.21.15.64:6790","nonce":0}]},"addr":"172.21.15.64:6790/0","public_addr":"172.21.15.64:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-10T05:51:12.691 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 3 2024-02-10T05:51:12.985 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:12 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:12.986 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:12 smithi176 ceph-mon[30294]: from='client.? 172.21.15.176:0/3523230548' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:51:12.986 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:12 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:51:13.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:12 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:13.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:12 smithi064 ceph-mon[25185]: from='client.? 172.21.15.176:0/3523230548' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:51:13.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:12 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:51:13.256 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-10T05:51:13.256 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph config generate-minimal-conf 2024-02-10T05:51:13.493 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:51:13.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:13.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:51:13.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: mon.a calling monitor election 2024-02-10T05:51:13.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: mon.b calling monitor election 2024-02-10T05:51:13.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:51:13.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:51:13.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:51:13.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:13.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:51:13.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:51:13.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:13.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:51:13.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:51:13.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-10T05:51:13.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: monmap e3: 3 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0],c=[v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:51:13.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: fsmap 2024-02-10T05:51:13.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:51:13.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: mgrmap e13: y(active, since 47s) 2024-02-10T05:51:13.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-02-10T05:51:13.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-02-10T05:51:13.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-02-10T05:51:13.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: mon.c (rank 2) addr [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] is down (out of quorum) 2024-02-10T05:51:13.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:13.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:13.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:13.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:13.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:13.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: from='client.? 172.21.15.176:0/3523230548' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:51:13.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:13 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:51:14.866 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: mon.c calling monitor election 2024-02-10T05:51:14.866 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-10T05:51:14.866 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: Updating smithi176:/etc/ceph/ceph.conf 2024-02-10T05:51:14.866 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: mon.c calling monitor election 2024-02-10T05:51:14.866 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: mon.b calling monitor election 2024-02-10T05:51:14.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: mon.a calling monitor election 2024-02-10T05:51:14.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-10T05:51:14.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: monmap e3: 3 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0],c=[v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:51:14.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: fsmap 2024-02-10T05:51:14.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:51:14.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: mgrmap e13: y(active, since 49s) 2024-02-10T05:51:14.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-02-10T05:51:14.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: Cluster is now healthy 2024-02-10T05:51:14.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: overall HEALTH_OK 2024-02-10T05:51:14.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.868 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.868 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.868 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.868 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.868 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.868 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.868 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:51:14.868 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:51:14.868 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:14.868 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: mon.c calling monitor election 2024-02-10T05:51:14.869 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-10T05:51:14.869 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: Updating smithi176:/etc/ceph/ceph.conf 2024-02-10T05:51:14.869 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: mon.c calling monitor election 2024-02-10T05:51:14.869 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: mon.b calling monitor election 2024-02-10T05:51:14.869 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: mon.a calling monitor election 2024-02-10T05:51:14.869 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-10T05:51:14.869 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: monmap e3: 3 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0],c=[v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:51:14.869 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: fsmap 2024-02-10T05:51:14.869 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:51:14.869 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: mgrmap e13: y(active, since 49s) 2024-02-10T05:51:14.869 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-02-10T05:51:14.870 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: Cluster is now healthy 2024-02-10T05:51:14.870 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: overall HEALTH_OK 2024-02-10T05:51:14.870 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.870 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.870 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.870 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.870 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.870 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.870 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.870 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.870 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.870 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.870 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:51:14.871 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:51:14.871 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:14 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:14.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: mon.c calling monitor election 2024-02-10T05:51:14.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-10T05:51:14.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: Updating smithi176:/etc/ceph/ceph.conf 2024-02-10T05:51:14.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: mon.c calling monitor election 2024-02-10T05:51:14.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: mon.b calling monitor election 2024-02-10T05:51:14.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: mon.a calling monitor election 2024-02-10T05:51:14.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-10T05:51:14.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: monmap e3: 3 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0],c=[v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:51:14.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: fsmap 2024-02-10T05:51:14.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:51:14.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: mgrmap e13: y(active, since 49s) 2024-02-10T05:51:14.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-02-10T05:51:14.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: Cluster is now healthy 2024-02-10T05:51:14.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: overall HEALTH_OK 2024-02-10T05:51:14.990 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.990 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.990 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.990 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.990 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.990 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.990 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.990 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.990 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.990 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:14.990 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:51:14.990 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:51:14.990 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:14 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:15.312 INFO:teuthology.orchestra.run.smithi064.stdout:# minimal ceph.conf for 00e86dbe-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:51:15.312 INFO:teuthology.orchestra.run.smithi064.stdout:[global] 2024-02-10T05:51:15.312 INFO:teuthology.orchestra.run.smithi064.stdout: fsid = 00e86dbe-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:51:15.313 INFO:teuthology.orchestra.run.smithi064.stdout: mon_host = [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] [v2:172.21.15.64:3301/0,v1:172.21.15.64:6790/0] 2024-02-10T05:51:15.846 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-10T05:51:15.847 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T05:51:15.847 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T05:51:15.876 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T05:51:15.876 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:51:15.939 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:15 smithi064 ceph-mon[25185]: Updating smithi176:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:51:15.939 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:15 smithi064 ceph-mon[25185]: Updating smithi064:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:51:15.939 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:15 smithi064 ceph-mon[25185]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:15.940 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:15 smithi064 ceph-mon[25185]: Reconfiguring mon.a (unknown last config time)... 2024-02-10T05:51:15.940 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:15 smithi064 ceph-mon[25185]: Reconfiguring daemon mon.a on smithi064 2024-02-10T05:51:15.940 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:15 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:51:15.940 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:15 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/1015702122' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:15.940 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:15 smithi064 ceph-mon[38958]: Updating smithi176:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:51:15.940 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:15 smithi064 ceph-mon[38958]: Updating smithi064:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:51:15.940 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:15 smithi064 ceph-mon[38958]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:15.940 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:15 smithi064 ceph-mon[38958]: Reconfiguring mon.a (unknown last config time)... 2024-02-10T05:51:15.940 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:15 smithi064 ceph-mon[38958]: Reconfiguring daemon mon.a on smithi064 2024-02-10T05:51:15.940 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:15 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:51:15.941 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:15 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/1015702122' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:15.943 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-10T05:51:15.943 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T05:51:15.972 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:15 smithi176 ceph-mon[30294]: Updating smithi176:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:51:15.972 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:15 smithi176 ceph-mon[30294]: Updating smithi064:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:51:15.972 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:15 smithi176 ceph-mon[30294]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:15.973 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:15 smithi176 ceph-mon[30294]: Reconfiguring mon.a (unknown last config time)... 2024-02-10T05:51:15.973 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:15 smithi176 ceph-mon[30294]: Reconfiguring daemon mon.a on smithi064 2024-02-10T05:51:15.973 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:15 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:51:15.973 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:15 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/1015702122' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:15.978 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-10T05:51:15.978 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:51:16.048 INFO:tasks.cephadm:Adding mgr.y on smithi064 2024-02-10T05:51:16.048 INFO:tasks.cephadm:Adding mgr.x on smithi176 2024-02-10T05:51:16.048 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph orch apply mgr '2;smithi064=y;smithi176=x' 2024-02-10T05:51:16.320 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:51:17.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:17 smithi064 ceph-mon[38958]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:17.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:17 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:17.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:17 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:17.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:17 smithi064 ceph-mon[38958]: Reconfiguring mon.c (monmap changed)... 2024-02-10T05:51:17.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:17 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:51:17.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:17 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:51:17.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:17 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:17.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:17 smithi064 ceph-mon[38958]: Reconfiguring daemon mon.c on smithi064 2024-02-10T05:51:17.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:17 smithi064 ceph-mon[25185]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:17.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:17 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:17.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:17 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:17.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:17 smithi064 ceph-mon[25185]: Reconfiguring mon.c (monmap changed)... 2024-02-10T05:51:17.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:17 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:51:17.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:17 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:51:17.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:17 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:17.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:17 smithi064 ceph-mon[25185]: Reconfiguring daemon mon.c on smithi064 2024-02-10T05:51:17.907 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled mgr update... 2024-02-10T05:51:17.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:17 smithi176 ceph-mon[30294]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:17.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:17 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:17.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:17 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:17.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:17 smithi176 ceph-mon[30294]: Reconfiguring mon.c (monmap changed)... 2024-02-10T05:51:17.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:17 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:51:17.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:17 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:51:17.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:17 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:17.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:17 smithi176 ceph-mon[30294]: Reconfiguring daemon mon.c on smithi064 2024-02-10T05:51:18.537 DEBUG:teuthology.orchestra.run.smithi176:mgr.x> sudo journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@mgr.x.service 2024-02-10T05:51:18.539 INFO:tasks.cephadm:Deploying OSDs... 2024-02-10T05:51:18.540 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T05:51:18.540 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T05:51:18.555 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-02-10T05:51:18.556 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/nvme1n1 2024-02-10T05:51:18.611 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/nvme1n1 2024-02-10T05:51:18.611 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-10T05:51:18.611 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-02-10T05:51:18.611 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-10T05:51:18.611 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-10T05:51:18.611 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-02-10 05:50:35.982720607 +0000 2024-02-10T05:51:18.611 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-02-10 05:48:21.326324016 +0000 2024-02-10T05:51:18.611 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-02-10 05:48:21.326324016 +0000 2024-02-10T05:51:18.612 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-02-10 05:48:21.315324230 +0000 2024-02-10T05:51:18.612 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-02-10T05:51:18.678 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-02-10T05:51:18.678 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-02-10T05:51:18.678 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000273453 s, 1.9 MB/s 2024-02-10T05:51:18.679 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-02-10T05:51:18.735 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/nvme2n1 2024-02-10T05:51:18.789 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/nvme2n1 2024-02-10T05:51:18.790 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-10T05:51:18.790 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 774 Links: 1 Device type: 103,4 2024-02-10T05:51:18.790 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-10T05:51:18.790 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-10T05:51:18.790 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-02-10 05:50:35.993720393 +0000 2024-02-10T05:51:18.790 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-02-10 05:48:21.475321120 +0000 2024-02-10T05:51:18.790 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-02-10 05:48:21.475321120 +0000 2024-02-10T05:51:18.790 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-02-10 05:48:21.464321334 +0000 2024-02-10T05:51:18.790 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-02-10T05:51:18.856 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-02-10T05:51:18.856 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-02-10T05:51:18.856 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000231158 s, 2.2 MB/s 2024-02-10T05:51:18.857 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-02-10T05:51:18.914 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/nvme3n1 2024-02-10T05:51:18.969 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/nvme3n1 2024-02-10T05:51:18.969 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-10T05:51:18.969 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-02-10T05:51:18.970 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-10T05:51:18.970 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-10T05:51:18.970 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-02-10 05:50:36.004720180 +0000 2024-02-10T05:51:18.970 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-02-10 05:48:21.618318341 +0000 2024-02-10T05:51:18.970 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-02-10 05:48:21.618318341 +0000 2024-02-10T05:51:18.970 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-02-10 05:48:21.607318555 +0000 2024-02-10T05:51:18.971 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-02-10T05:51:19.022 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:18 smithi064 ceph-mon[25185]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi064=y;smithi176=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:19.022 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:18 smithi064 ceph-mon[25185]: Saving service mgr spec with placement smithi064=y;smithi176=x;count:2 2024-02-10T05:51:19.023 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:18 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:19.023 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:18 smithi064 ceph-mon[25185]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:19.023 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:18 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:19.023 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:18 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:19.023 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:18 smithi064 ceph-mon[25185]: Reconfiguring mon.b (monmap changed)... 2024-02-10T05:51:19.023 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:18 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:51:19.023 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:18 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:51:19.023 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:18 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:19.024 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:18 smithi064 ceph-mon[25185]: Reconfiguring daemon mon.b on smithi176 2024-02-10T05:51:19.024 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:18 smithi064 ceph-mon[38958]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi064=y;smithi176=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:19.024 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:18 smithi064 ceph-mon[38958]: Saving service mgr spec with placement smithi064=y;smithi176=x;count:2 2024-02-10T05:51:19.024 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:18 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:19.024 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:18 smithi064 ceph-mon[38958]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:19.024 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:18 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:19.024 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:18 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:19.024 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:18 smithi064 ceph-mon[38958]: Reconfiguring mon.b (monmap changed)... 2024-02-10T05:51:19.024 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:18 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:51:19.024 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:18 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:51:19.025 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:18 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:19.025 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:18 smithi064 ceph-mon[38958]: Reconfiguring daemon mon.b on smithi176 2024-02-10T05:51:19.035 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-02-10T05:51:19.035 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-02-10T05:51:19.035 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000236395 s, 2.2 MB/s 2024-02-10T05:51:19.036 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-02-10T05:51:19.091 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/nvme4n1 2024-02-10T05:51:19.146 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/nvme4n1 2024-02-10T05:51:19.146 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-10T05:51:19.146 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-02-10T05:51:19.146 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-10T05:51:19.146 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-10T05:51:19.146 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-02-10 05:50:36.015719966 +0000 2024-02-10T05:51:19.146 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-02-10 05:48:21.761315562 +0000 2024-02-10T05:51:19.146 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-02-10 05:48:21.761315562 +0000 2024-02-10T05:51:19.146 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-02-10 05:48:21.749315795 +0000 2024-02-10T05:51:19.147 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-02-10T05:51:19.204 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:18 smithi176 ceph-mon[30294]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi064=y;smithi176=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:19.204 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:18 smithi176 ceph-mon[30294]: Saving service mgr spec with placement smithi064=y;smithi176=x;count:2 2024-02-10T05:51:19.204 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:18 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:19.204 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:18 smithi176 ceph-mon[30294]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:19.204 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:18 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:19.204 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:18 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:19.204 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:18 smithi176 ceph-mon[30294]: Reconfiguring mon.b (monmap changed)... 2024-02-10T05:51:19.204 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:18 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:51:19.204 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:18 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:51:19.204 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:18 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:19.204 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:18 smithi176 ceph-mon[30294]: Reconfiguring daemon mon.b on smithi176 2024-02-10T05:51:19.211 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-02-10T05:51:19.212 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-02-10T05:51:19.212 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000191261 s, 2.7 MB/s 2024-02-10T05:51:19.213 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-02-10T05:51:19.269 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-10T05:51:19.269 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T05:51:19.286 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-02-10T05:51:19.286 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/nvme1n1 2024-02-10T05:51:19.341 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/nvme1n1 2024-02-10T05:51:19.341 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-10T05:51:19.341 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-02-10T05:51:19.341 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-10T05:51:19.341 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-10T05:51:19.341 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-02-10 05:50:50.359986536 +0000 2024-02-10T05:51:19.341 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-02-10 05:48:22.974199348 +0000 2024-02-10T05:51:19.342 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-02-10 05:48:22.974199348 +0000 2024-02-10T05:51:19.342 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-02-10 05:48:22.961199464 +0000 2024-02-10T05:51:19.342 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-02-10T05:51:19.407 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-02-10T05:51:19.407 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-02-10T05:51:19.408 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000208649 s, 2.5 MB/s 2024-02-10T05:51:19.408 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-02-10T05:51:19.465 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/nvme2n1 2024-02-10T05:51:19.520 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/nvme2n1 2024-02-10T05:51:19.520 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-10T05:51:19.520 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 774 Links: 1 Device type: 103,4 2024-02-10T05:51:19.520 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-10T05:51:19.520 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-10T05:51:19.521 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-02-10 05:50:50.370986438 +0000 2024-02-10T05:51:19.521 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-02-10 05:48:23.127197983 +0000 2024-02-10T05:51:19.521 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-02-10 05:48:23.127197983 +0000 2024-02-10T05:51:19.521 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-02-10 05:48:23.116198082 +0000 2024-02-10T05:51:19.521 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-02-10T05:51:19.586 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-02-10T05:51:19.586 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-02-10T05:51:19.586 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000252638 s, 2.0 MB/s 2024-02-10T05:51:19.587 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-02-10T05:51:19.643 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/nvme3n1 2024-02-10T05:51:19.698 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/nvme3n1 2024-02-10T05:51:19.698 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-10T05:51:19.698 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-02-10T05:51:19.698 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-10T05:51:19.698 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-10T05:51:19.699 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-02-10 05:50:50.381986341 +0000 2024-02-10T05:51:19.699 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-02-10 05:48:23.280196619 +0000 2024-02-10T05:51:19.699 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-02-10 05:48:23.280196619 +0000 2024-02-10T05:51:19.699 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-02-10 05:48:23.272196691 +0000 2024-02-10T05:51:19.699 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-02-10T05:51:19.764 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-02-10T05:51:19.765 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-02-10T05:51:19.765 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000231458 s, 2.2 MB/s 2024-02-10T05:51:19.766 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-02-10T05:51:19.822 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/nvme4n1 2024-02-10T05:51:19.877 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/nvme4n1 2024-02-10T05:51:19.877 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-10T05:51:19.877 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 798 Links: 1 Device type: 103,8 2024-02-10T05:51:19.877 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-10T05:51:19.877 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-10T05:51:19.877 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-02-10 05:50:50.391986252 +0000 2024-02-10T05:51:19.877 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-02-10 05:48:23.432195264 +0000 2024-02-10T05:51:19.877 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-02-10 05:48:23.432195264 +0000 2024-02-10T05:51:19.877 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-02-10 05:48:23.420195371 +0000 2024-02-10T05:51:19.878 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-02-10T05:51:19.943 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-02-10T05:51:19.943 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-02-10T05:51:19.943 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000196458 s, 2.6 MB/s 2024-02-10T05:51:19.944 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-02-10T05:51:19.999 INFO:tasks.cephadm:Deploying osd.0 on smithi064 with /dev/nvme4n1... 2024-02-10T05:51:20.000 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-02-10T05:51:20.236 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:51:21.089 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:20 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:21.089 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:20 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:21.089 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:20 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:21.089 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:20 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:21.089 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:20 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:51:21.089 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:20 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:21.089 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:20 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:51:21.090 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:20 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T05:51:21.090 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:20 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:51:21.090 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:20 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:21.090 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:20 smithi064 ceph-mon[38958]: Deploying daemon mgr.x on smithi176 2024-02-10T05:51:21.090 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:20 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:21.090 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:20 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:21.090 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:20 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:21.090 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:20 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:21.090 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:20 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:51:21.091 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:20 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:21.091 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:20 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:51:21.091 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:20 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T05:51:21.091 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:20 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:51:21.091 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:20 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:21.091 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:20 smithi064 ceph-mon[25185]: Deploying daemon mgr.x on smithi176 2024-02-10T05:51:21.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:20 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:21.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:20 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:21.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:20 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:21.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:20 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:21.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:20 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:51:21.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:20 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:21.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:20 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:51:21.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:20 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T05:51:21.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:20 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:51:21.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:20 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:21.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:20 smithi176 ceph-mon[30294]: Deploying daemon mgr.x on smithi176 2024-02-10T05:51:21.874 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/nvme4n1 2024-02-10T05:51:21.874 INFO:teuthology.orchestra.run.smithi064.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-02-10T05:51:21.874 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-02-10T05:51:21.874 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-02-10T05:51:21.874 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-02-10T05:51:21.874 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124279 s, 844 MB/s 2024-02-10T05:51:21.874 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-02-10T05:51:21.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:21 smithi176 ceph-mon[30294]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:22.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:21 smithi064 ceph-mon[25185]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:22.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:21 smithi064 ceph-mon[38958]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:22.317 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:22 smithi176 ceph-mgr[32155]: -- 172.21.15.176:0/3474232407 <== mon.1 v2:172.21.15.176:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556656c581a0 con 0x556656c2f000 2024-02-10T05:51:22.409 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi064:/dev/nvme4n1 2024-02-10T05:51:22.664 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:51:22.738 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:22 smithi176 bash[32136]: 0404dd01867ca4247fa872a9037d4ca5b48b2969383d506a9893f8771588909a 2024-02-10T05:51:22.738 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:22 smithi176 systemd[1]: Started Ceph mgr.x for 00e86dbe-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:51:23.125 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:22 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:22.814+0000 7ff4b86a3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:51:23.126 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:22 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:22.911+0000 7ff4b86a3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:51:23.691 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:23 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:23.597+0000 7ff4b86a3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:51:23.692 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:23 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:23.691+0000 7ff4b86a3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:51:23.692 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:23 smithi176 ceph-mon[30294]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:23.692 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:23 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:23.692 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:23 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:23.692 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:23 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:23.692 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:23 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:23.692 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:23 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:23.727 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:23 smithi064 ceph-mon[25185]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:23.727 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:23 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:23.727 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:23 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:23.727 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:23 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:23.727 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:23 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:23.727 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:23 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:23.727 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:23 smithi064 ceph-mon[38958]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:23.727 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:23 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:23.727 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:23 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:23.727 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:23 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:23.728 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:23 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:23.728 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:23 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:23.942 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:23 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:23.843+0000 7ff4b86a3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:51:23.943 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:23 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:23.942+0000 7ff4b86a3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:51:24.238 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:24 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:24.027+0000 7ff4b86a3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:51:24.581 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:24 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:24.291+0000 7ff4b86a3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T05:51:24.581 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:24 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:24.400+0000 7ff4b86a3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:51:24.581 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:24 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:24.486+0000 7ff4b86a3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:51:24.581 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:24 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:24.580+0000 7ff4b86a3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:51:24.988 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:24 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:24.756+0000 7ff4b86a3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:51:24.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:24 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:24.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:24 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:24.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:24 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:24.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:24 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:51:24.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:24 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:24.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:24 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:51:24.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:24 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:51:24.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:24 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:24.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:24 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:24.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:24 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:51:24.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:24 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:51:24.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:24 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:25.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:24 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:25.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:24 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:25.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:24 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:25.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:24 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:51:25.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:24 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:25.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:24 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:51:25.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:24 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:51:25.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:24 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:25.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:24 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:25.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:24 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:51:25.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:24 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:51:25.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:24 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:25.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:24 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:25.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:24 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:25.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:24 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:25.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:24 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:51:25.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:24 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:25.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:24 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:51:25.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:24 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:51:25.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:24 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:25.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:24 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:25.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:24 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:51:25.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:24 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:51:25.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:24 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:26.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:25 smithi064 ceph-mon[25185]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:26.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:25 smithi064 ceph-mon[25185]: Reconfiguring mgr.y (unknown last config time)... 2024-02-10T05:51:26.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:25 smithi064 ceph-mon[25185]: Reconfiguring daemon mgr.y on smithi064 2024-02-10T05:51:26.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:25 smithi064 ceph-mon[25185]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:26.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:25 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:26.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:25 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:26.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:25 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:26.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:25 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:26.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:25 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:51:26.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:25 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:26.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:25 smithi064 ceph-mon[38958]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:26.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:25 smithi064 ceph-mon[38958]: Reconfiguring mgr.y (unknown last config time)... 2024-02-10T05:51:26.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:25 smithi064 ceph-mon[38958]: Reconfiguring daemon mgr.y on smithi064 2024-02-10T05:51:26.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:25 smithi064 ceph-mon[38958]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:26.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:25 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:26.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:25 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:26.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:25 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:26.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:25 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:26.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:25 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:51:26.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:25 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:26.213 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:25 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:25.746+0000 7ff4b86a3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:51:26.213 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:25 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:25.879+0000 7ff4b86a3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:51:26.213 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:26 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:26.212+0000 7ff4b86a3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:51:26.213 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:25 smithi176 ceph-mon[30294]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:26.213 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:25 smithi176 ceph-mon[30294]: Reconfiguring mgr.y (unknown last config time)... 2024-02-10T05:51:26.213 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:25 smithi176 ceph-mon[30294]: Reconfiguring daemon mgr.y on smithi064 2024-02-10T05:51:26.214 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:25 smithi176 ceph-mon[30294]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:26.214 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:25 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:26.214 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:25 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:26.214 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:25 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:26.214 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:25 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:26.214 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:25 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:51:26.214 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:25 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:26.488 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:26 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:26.302+0000 7ff4b86a3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:51:26.967 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:26 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:26.495+0000 7ff4b86a3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:51:26.967 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:26 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:26.680+0000 7ff4b86a3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:51:27.227 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:26 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:26.966+0000 7ff4b86a3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:51:27.227 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:27 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:27.227+0000 7ff4b86a3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:51:27.488 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:27 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:27.462+0000 7ff4b86a3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:51:28.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:27 smithi064 ceph-mon[25185]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:28.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:27 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/2519497134' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32df1361-a86f-44aa-bc33-d0b1e3fc140b"}]: dispatch 2024-02-10T05:51:28.128 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:27 smithi064 ceph-mon[25185]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32df1361-a86f-44aa-bc33-d0b1e3fc140b"}]: dispatch 2024-02-10T05:51:28.128 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:27 smithi064 ceph-mon[25185]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32df1361-a86f-44aa-bc33-d0b1e3fc140b"}]': finished 2024-02-10T05:51:28.128 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:27 smithi064 ceph-mon[25185]: osdmap e5: 1 total, 0 up, 1 in 2024-02-10T05:51:28.128 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:27 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:51:28.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:27 smithi064 ceph-mon[38958]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:28.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:27 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/2519497134' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32df1361-a86f-44aa-bc33-d0b1e3fc140b"}]: dispatch 2024-02-10T05:51:28.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:27 smithi064 ceph-mon[38958]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32df1361-a86f-44aa-bc33-d0b1e3fc140b"}]: dispatch 2024-02-10T05:51:28.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:27 smithi064 ceph-mon[38958]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32df1361-a86f-44aa-bc33-d0b1e3fc140b"}]': finished 2024-02-10T05:51:28.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:27 smithi064 ceph-mon[38958]: osdmap e5: 1 total, 0 up, 1 in 2024-02-10T05:51:28.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:27 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:51:28.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:27 smithi176 ceph-mon[30294]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:28.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:27 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/2519497134' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32df1361-a86f-44aa-bc33-d0b1e3fc140b"}]: dispatch 2024-02-10T05:51:28.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:27 smithi176 ceph-mon[30294]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32df1361-a86f-44aa-bc33-d0b1e3fc140b"}]: dispatch 2024-02-10T05:51:28.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:27 smithi176 ceph-mon[30294]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32df1361-a86f-44aa-bc33-d0b1e3fc140b"}]': finished 2024-02-10T05:51:28.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:27 smithi176 ceph-mon[30294]: osdmap e5: 1 total, 0 up, 1 in 2024-02-10T05:51:28.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:27 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:51:28.738 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:28 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:28.265+0000 7ff4b86a3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:51:29.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:28 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/158262031' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:51:29.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:28 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/158262031' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:51:29.152 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:28 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/158262031' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:51:29.152 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:28 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:28.835+0000 7ff4b86a3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:51:29.152 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:28 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:28.936+0000 7ff4b86a3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:51:29.152 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:51:29 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:51:29.151+0000 7ff4b86a3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:51:30.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:29 smithi064 ceph-mon[38958]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:30.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:29 smithi064 ceph-mon[38958]: Standby manager daemon x started 2024-02-10T05:51:30.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:29 smithi064 ceph-mon[38958]: from='mgr.? 172.21.15.176:0/4160923479' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T05:51:30.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:29 smithi064 ceph-mon[38958]: from='mgr.? 172.21.15.176:0/4160923479' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:51:30.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:29 smithi064 ceph-mon[38958]: from='mgr.? 172.21.15.176:0/4160923479' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T05:51:30.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:29 smithi064 ceph-mon[38958]: from='mgr.? 172.21.15.176:0/4160923479' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:51:30.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:29 smithi064 ceph-mon[25185]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:30.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:29 smithi064 ceph-mon[25185]: Standby manager daemon x started 2024-02-10T05:51:30.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:29 smithi064 ceph-mon[25185]: from='mgr.? 172.21.15.176:0/4160923479' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T05:51:30.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:29 smithi064 ceph-mon[25185]: from='mgr.? 172.21.15.176:0/4160923479' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:51:30.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:29 smithi064 ceph-mon[25185]: from='mgr.? 172.21.15.176:0/4160923479' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T05:51:30.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:29 smithi064 ceph-mon[25185]: from='mgr.? 172.21.15.176:0/4160923479' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:51:30.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:29 smithi176 ceph-mon[30294]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:30.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:29 smithi176 ceph-mon[30294]: Standby manager daemon x started 2024-02-10T05:51:30.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:29 smithi176 ceph-mon[30294]: from='mgr.? 172.21.15.176:0/4160923479' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T05:51:30.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:29 smithi176 ceph-mon[30294]: from='mgr.? 172.21.15.176:0/4160923479' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:51:30.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:29 smithi176 ceph-mon[30294]: from='mgr.? 172.21.15.176:0/4160923479' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T05:51:30.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:29 smithi176 ceph-mon[30294]: from='mgr.? 172.21.15.176:0/4160923479' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:51:31.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:30 smithi064 ceph-mon[25185]: mgrmap e14: y(active, since 65s), standbys: x 2024-02-10T05:51:31.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:30 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T05:51:31.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:30 smithi064 ceph-mon[38958]: mgrmap e14: y(active, since 65s), standbys: x 2024-02-10T05:51:31.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:30 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T05:51:31.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:30 smithi176 ceph-mon[30294]: mgrmap e14: y(active, since 65s), standbys: x 2024-02-10T05:51:31.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:30 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T05:51:32.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:31 smithi064 ceph-mon[25185]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:32.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:31 smithi064 ceph-mon[38958]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:32.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:31 smithi176 ceph-mon[30294]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:33.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:32 smithi176 ceph-mon[30294]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:33.267 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:32 smithi064 ceph-mon[25185]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:33.267 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:32 smithi064 ceph-mon[38958]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:35.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:34 smithi176 ceph-mon[30294]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:35.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:34 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T05:51:35.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:34 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:35.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:34 smithi176 ceph-mon[30294]: Deploying daemon osd.0 on smithi064 2024-02-10T05:51:35.308 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:34 smithi064 ceph-mon[25185]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:35.308 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:34 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T05:51:35.309 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:34 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:35.309 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:34 smithi064 ceph-mon[25185]: Deploying daemon osd.0 on smithi064 2024-02-10T05:51:35.309 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:34 smithi064 ceph-mon[38958]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:35.309 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:34 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T05:51:35.309 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:34 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:35.309 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:34 smithi064 ceph-mon[38958]: Deploying daemon osd.0 on smithi064 2024-02-10T05:51:37.815 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:37 smithi064 ceph-mon[25185]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:37.815 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:37 smithi064 ceph-mon[38958]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:37.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:37 smithi176 ceph-mon[30294]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:39.849 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:39 smithi064 ceph-mon[25185]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:39.850 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:39 smithi064 ceph-mon[38958]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:39.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:39 smithi176 ceph-mon[30294]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:41.348 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:41 smithi064 ceph-mon[25185]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:41.349 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:41 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:41.349 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:41 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:41.349 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:41 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:41.349 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:41 smithi064 ceph-mon[38958]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:41.349 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:41 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:41.349 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:41 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:41.349 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:41 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:41.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:41 smithi176 ceph-mon[30294]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:41.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:41 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:41.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:41 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:41.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:41 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:43.763 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 0 on host 'smithi064' 2024-02-10T05:51:44.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:43 smithi064 ceph-mon[25185]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:44.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:43 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:44.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:43 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:44.065 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:43 smithi064 ceph-mon[25185]: from='osd.0 [v2:172.21.15.64:6802/3211593074,v1:172.21.15.64:6803/3211593074]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T05:51:44.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:43 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:44.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:43 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:51:44.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:43 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:44.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:43 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:44.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:43 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:44.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:43 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:44.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:43 smithi064 ceph-mon[38958]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:44.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:43 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:44.066 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:43 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:44.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:43 smithi064 ceph-mon[38958]: from='osd.0 [v2:172.21.15.64:6802/3211593074,v1:172.21.15.64:6803/3211593074]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T05:51:44.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:43 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:44.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:43 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:51:44.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:43 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:44.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:43 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:44.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:43 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:44.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:43 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:44.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:43 smithi176 ceph-mon[30294]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:44.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:43 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:44.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:43 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:44.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:43 smithi176 ceph-mon[30294]: from='osd.0 [v2:172.21.15.64:6802/3211593074,v1:172.21.15.64:6803/3211593074]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T05:51:44.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:43 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:44.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:43 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:51:44.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:43 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:44.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:43 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:44.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:43 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:44.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:43 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:51:44.373 DEBUG:teuthology.orchestra.run.smithi064:osd.0> sudo journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.0.service 2024-02-10T05:51:44.375 INFO:tasks.cephadm:Deploying osd.1 on smithi064 with /dev/nvme3n1... 2024-02-10T05:51:44.375 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/nvme3n1 2024-02-10T05:51:45.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:44 smithi064 ceph-mon[25185]: from='osd.0 [v2:172.21.15.64:6802/3211593074,v1:172.21.15.64:6803/3211593074]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T05:51:45.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:44 smithi064 ceph-mon[25185]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T05:51:45.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:44 smithi064 ceph-mon[25185]: from='osd.0 [v2:172.21.15.64:6802/3211593074,v1:172.21.15.64:6803/3211593074]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T05:51:45.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:44 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:51:45.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:44 smithi064 ceph-mon[38958]: from='osd.0 [v2:172.21.15.64:6802/3211593074,v1:172.21.15.64:6803/3211593074]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T05:51:45.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:44 smithi064 ceph-mon[38958]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T05:51:45.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:44 smithi064 ceph-mon[38958]: from='osd.0 [v2:172.21.15.64:6802/3211593074,v1:172.21.15.64:6803/3211593074]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T05:51:45.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:44 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:51:45.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:44 smithi176 ceph-mon[30294]: from='osd.0 [v2:172.21.15.64:6802/3211593074,v1:172.21.15.64:6803/3211593074]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T05:51:45.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:44 smithi176 ceph-mon[30294]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T05:51:45.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:44 smithi176 ceph-mon[30294]: from='osd.0 [v2:172.21.15.64:6802/3211593074,v1:172.21.15.64:6803/3211593074]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T05:51:45.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:44 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:51:45.455 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:51:45 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-0[42822]: 2024-02-10T05:51:45.256+0000 7f41f24d2700 -1 osd.0 0 waiting for initial osdmap 2024-02-10T05:51:45.455 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:51:45 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-0[42822]: 2024-02-10T05:51:45.258+0000 7f41ea2bd700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:51:45.540 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:51:46.094 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:45 smithi064 ceph-mon[25185]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:46.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:45 smithi064 ceph-mon[25185]: from='osd.0 [v2:172.21.15.64:6802/3211593074,v1:172.21.15.64:6803/3211593074]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-10T05:51:46.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:45 smithi064 ceph-mon[25185]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T05:51:46.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:45 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:51:46.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:45 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:51:46.095 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:45 smithi064 ceph-mon[25185]: from='osd.0 [v2:172.21.15.64:6802/3211593074,v1:172.21.15.64:6803/3211593074]' entity='osd.0' 2024-02-10T05:51:46.095 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:45 smithi064 ceph-mon[38958]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:46.095 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:45 smithi064 ceph-mon[38958]: from='osd.0 [v2:172.21.15.64:6802/3211593074,v1:172.21.15.64:6803/3211593074]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-10T05:51:46.095 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:45 smithi064 ceph-mon[38958]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T05:51:46.095 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:45 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:51:46.095 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:45 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:51:46.096 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:45 smithi064 ceph-mon[38958]: from='osd.0 [v2:172.21.15.64:6802/3211593074,v1:172.21.15.64:6803/3211593074]' entity='osd.0' 2024-02-10T05:51:46.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:45 smithi176 ceph-mon[30294]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:46.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:45 smithi176 ceph-mon[30294]: from='osd.0 [v2:172.21.15.64:6802/3211593074,v1:172.21.15.64:6803/3211593074]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-10T05:51:46.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:45 smithi176 ceph-mon[30294]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T05:51:46.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:45 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:51:46.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:45 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:51:46.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:45 smithi176 ceph-mon[30294]: from='osd.0 [v2:172.21.15.64:6802/3211593074,v1:172.21.15.64:6803/3211593074]' entity='osd.0' 2024-02-10T05:51:47.049 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:46 smithi064 ceph-mon[25185]: purged_snaps scrub starts 2024-02-10T05:51:47.049 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:46 smithi064 ceph-mon[25185]: purged_snaps scrub ok 2024-02-10T05:51:47.049 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:46 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:51:47.049 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:46 smithi064 ceph-mon[25185]: osd.0 [v2:172.21.15.64:6802/3211593074,v1:172.21.15.64:6803/3211593074] boot 2024-02-10T05:51:47.049 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:46 smithi064 ceph-mon[25185]: osdmap e8: 1 total, 1 up, 1 in 2024-02-10T05:51:47.049 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:46 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:51:47.049 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:46 smithi064 ceph-mon[38958]: purged_snaps scrub starts 2024-02-10T05:51:47.050 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:46 smithi064 ceph-mon[38958]: purged_snaps scrub ok 2024-02-10T05:51:47.050 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:46 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:51:47.050 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:46 smithi064 ceph-mon[38958]: osd.0 [v2:172.21.15.64:6802/3211593074,v1:172.21.15.64:6803/3211593074] boot 2024-02-10T05:51:47.050 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:46 smithi064 ceph-mon[38958]: osdmap e8: 1 total, 1 up, 1 in 2024-02-10T05:51:47.050 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:46 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:51:47.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:46 smithi176 ceph-mon[30294]: purged_snaps scrub starts 2024-02-10T05:51:47.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:46 smithi176 ceph-mon[30294]: purged_snaps scrub ok 2024-02-10T05:51:47.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:46 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:51:47.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:46 smithi176 ceph-mon[30294]: osd.0 [v2:172.21.15.64:6802/3211593074,v1:172.21.15.64:6803/3211593074] boot 2024-02-10T05:51:47.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:46 smithi176 ceph-mon[30294]: osdmap e8: 1 total, 1 up, 1 in 2024-02-10T05:51:47.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:46 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:51:47.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:47 smithi064 ceph-mon[25185]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:47.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:47.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:47.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:51:47.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:47.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:47.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:51:47.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:47.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:47 smithi064 ceph-mon[38958]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:47.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:47 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:47.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:47 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:47.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:47 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:51:47.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:47 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:47.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:47 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:47.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:47 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:51:47.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:47 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:48.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:47 smithi176 ceph-mon[30294]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:51:48.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:47 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:48.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:47 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:48.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:47 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:51:48.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:47 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:48.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:47 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:48.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:47 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:51:48.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:47 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:51:48.458 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/nvme3n1 2024-02-10T05:51:48.458 INFO:teuthology.orchestra.run.smithi064.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-02-10T05:51:48.458 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme3n1 bs=1M count=10 conv=fsync 2024-02-10T05:51:48.458 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-02-10T05:51:48.458 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-02-10T05:51:48.458 INFO:teuthology.orchestra.run.smithi064.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125749 s, 834 MB/s 2024-02-10T05:51:48.458 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-02-10T05:51:49.084 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:48 smithi064 ceph-mon[25185]: Detected new or changed devices on smithi064 2024-02-10T05:51:49.084 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:48 smithi064 ceph-mon[25185]: Adjusting osd_memory_target on smithi064 to 16064M 2024-02-10T05:51:49.085 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:48 smithi064 ceph-mon[25185]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T05:51:49.085 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:48 smithi064 ceph-mon[38958]: Detected new or changed devices on smithi064 2024-02-10T05:51:49.085 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:48 smithi064 ceph-mon[38958]: Adjusting osd_memory_target on smithi064 to 16064M 2024-02-10T05:51:49.085 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:48 smithi064 ceph-mon[38958]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T05:51:49.109 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi064:/dev/nvme3n1 2024-02-10T05:51:49.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:48 smithi176 ceph-mon[30294]: Detected new or changed devices on smithi064 2024-02-10T05:51:49.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:48 smithi176 ceph-mon[30294]: Adjusting osd_memory_target on smithi064 to 16064M 2024-02-10T05:51:49.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:48 smithi176 ceph-mon[30294]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T05:51:49.350 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:51:50.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:49 smithi064 ceph-mon[38958]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:51:50.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:49 smithi064 ceph-mon[25185]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:51:50.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:49 smithi176 ceph-mon[30294]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:51:51.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:51 smithi064 ceph-mon[25185]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:51:51.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:51 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:51:51.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:51 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:51:51.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:51 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:51.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:51 smithi064 ceph-mon[38958]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:51:51.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:51 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:51:51.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:51 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:51:51.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:51 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:52.121 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:51 smithi176 ceph-mon[30294]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:51:52.121 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:51 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:51:52.121 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:51 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:51:52.121 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:51 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:51:52.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:52 smithi176 ceph-mon[30294]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:53.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:52 smithi064 ceph-mon[25185]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:53.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:52 smithi064 ceph-mon[38958]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:51:54.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:53 smithi064 ceph-mon[25185]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:51:54.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:53 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/3804905836' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cfe981c9-406d-4da9-89ff-81dbbef91183"}]: dispatch 2024-02-10T05:51:54.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:53 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/3804905836' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cfe981c9-406d-4da9-89ff-81dbbef91183"}]': finished 2024-02-10T05:51:54.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:53 smithi064 ceph-mon[25185]: osdmap e10: 2 total, 1 up, 2 in 2024-02-10T05:51:54.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:53 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:51:54.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:53 smithi064 ceph-mon[38958]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:51:54.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:53 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/3804905836' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cfe981c9-406d-4da9-89ff-81dbbef91183"}]: dispatch 2024-02-10T05:51:54.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:53 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/3804905836' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cfe981c9-406d-4da9-89ff-81dbbef91183"}]': finished 2024-02-10T05:51:54.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:53 smithi064 ceph-mon[38958]: osdmap e10: 2 total, 1 up, 2 in 2024-02-10T05:51:54.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:53 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:51:54.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:53 smithi176 ceph-mon[30294]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:51:54.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:53 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/3804905836' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cfe981c9-406d-4da9-89ff-81dbbef91183"}]: dispatch 2024-02-10T05:51:54.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:53 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/3804905836' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cfe981c9-406d-4da9-89ff-81dbbef91183"}]': finished 2024-02-10T05:51:54.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:53 smithi176 ceph-mon[30294]: osdmap e10: 2 total, 1 up, 2 in 2024-02-10T05:51:54.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:53 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:51:56.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:55 smithi064 ceph-mon[25185]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:51:56.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:55 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/943894655' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:51:56.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:55 smithi064 ceph-mon[38958]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:51:56.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:55 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/943894655' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:51:56.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:55 smithi176 ceph-mon[30294]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:51:56.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:55 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/943894655' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:51:58.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:57 smithi064 ceph-mon[25185]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:51:58.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:57 smithi064 ceph-mon[38958]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:51:58.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:57 smithi176 ceph-mon[30294]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:00.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:51:59 smithi064 ceph-mon[25185]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:00.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:51:59 smithi064 ceph-mon[38958]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:00.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:51:59 smithi176 ceph-mon[30294]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:01.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:00 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T05:52:01.128 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:00 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:01.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:00 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T05:52:01.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:00 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:01.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:00 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T05:52:01.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:00 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:02.104 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:01 smithi064 ceph-mon[38958]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:02.104 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:01 smithi064 ceph-mon[38958]: Deploying daemon osd.1 on smithi064 2024-02-10T05:52:02.105 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:01 smithi064 ceph-mon[25185]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:02.105 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:01 smithi064 ceph-mon[25185]: Deploying daemon osd.1 on smithi064 2024-02-10T05:52:02.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:01 smithi176 ceph-mon[30294]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:02.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:01 smithi176 ceph-mon[30294]: Deploying daemon osd.1 on smithi064 2024-02-10T05:52:04.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:03 smithi064 ceph-mon[25185]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:04.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:03 smithi064 ceph-mon[38958]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:04.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:03 smithi176 ceph-mon[30294]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:06.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:05 smithi064 ceph-mon[25185]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:06.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:05 smithi064 ceph-mon[38958]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:06.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:05 smithi176 ceph-mon[30294]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:07.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:07 smithi064 ceph-mon[25185]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:07.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:07 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:07.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:07 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:07.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:07 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:07.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:07 smithi064 ceph-mon[38958]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:07.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:07 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:07.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:07 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:07.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:07 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:07.491 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:07 smithi176 ceph-mon[30294]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:07.491 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:07 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:07.491 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:07 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:07.491 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:07 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:09.816 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 1 on host 'smithi064' 2024-02-10T05:52:10.084 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:09 smithi064 ceph-mon[25185]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:10.084 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:09 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:10.085 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:09 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:10.085 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:09 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:10.085 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:09 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:52:10.085 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:09 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:10.085 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:09 smithi064 ceph-mon[25185]: from='osd.1 [v2:172.21.15.64:6810/614169574,v1:172.21.15.64:6811/614169574]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:52:10.085 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:09 smithi064 ceph-mon[25185]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:52:10.085 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:09 smithi064 ceph-mon[38958]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:10.085 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:09 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:10.085 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:09 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:10.085 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:09 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:10.085 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:09 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:52:10.086 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:09 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:10.086 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:09 smithi064 ceph-mon[38958]: from='osd.1 [v2:172.21.15.64:6810/614169574,v1:172.21.15.64:6811/614169574]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:52:10.086 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:09 smithi064 ceph-mon[38958]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:52:10.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:09 smithi176 ceph-mon[30294]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:10.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:09 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:10.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:09 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:10.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:09 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:10.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:09 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:52:10.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:09 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:10.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:09 smithi176 ceph-mon[30294]: from='osd.1 [v2:172.21.15.64:6810/614169574,v1:172.21.15.64:6811/614169574]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:52:10.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:09 smithi176 ceph-mon[30294]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:52:10.424 DEBUG:teuthology.orchestra.run.smithi064:osd.1> sudo journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.1.service 2024-02-10T05:52:10.426 INFO:tasks.cephadm:Deploying osd.2 on smithi064 with /dev/nvme2n1... 2024-02-10T05:52:10.426 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/nvme2n1 2024-02-10T05:52:11.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:10 smithi064 ceph-mon[25185]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T05:52:11.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:10 smithi064 ceph-mon[25185]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T05:52:11.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:10 smithi064 ceph-mon[25185]: from='osd.1 [v2:172.21.15.64:6810/614169574,v1:172.21.15.64:6811/614169574]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T05:52:11.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:10 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:52:11.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:10 smithi064 ceph-mon[25185]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T05:52:11.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:10 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:11.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:10 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:11.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:10 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:11.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:10 smithi064 ceph-mon[38958]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T05:52:11.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:10 smithi064 ceph-mon[38958]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T05:52:11.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:10 smithi064 ceph-mon[38958]: from='osd.1 [v2:172.21.15.64:6810/614169574,v1:172.21.15.64:6811/614169574]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T05:52:11.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:10 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:52:11.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:10 smithi064 ceph-mon[38958]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T05:52:11.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:10 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:11.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:10 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:11.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:10 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:11.128 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 05:52:10 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-1[50712]: 2024-02-10T05:52:10.801+0000 7f3ec8fd3700 -1 osd.1 0 waiting for initial osdmap 2024-02-10T05:52:11.128 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 05:52:10 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-1[50712]: 2024-02-10T05:52:10.806+0000 7f3ec4dc6700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:52:11.139 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:10 smithi176 ceph-mon[30294]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T05:52:11.140 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:10 smithi176 ceph-mon[30294]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T05:52:11.140 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:10 smithi176 ceph-mon[30294]: from='osd.1 [v2:172.21.15.64:6810/614169574,v1:172.21.15.64:6811/614169574]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T05:52:11.140 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:10 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:52:11.140 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:10 smithi176 ceph-mon[30294]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T05:52:11.140 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:10 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:11.140 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:10 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:11.140 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:10 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:11.450 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:52:11.825 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:11 smithi064 ceph-mon[25185]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:11.825 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:11 smithi064 ceph-mon[25185]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-10T05:52:11.825 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:11 smithi064 ceph-mon[25185]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T05:52:11.825 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:11 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:52:11.825 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:11 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:52:11.825 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:11 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:52:11.826 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:11 smithi064 ceph-mon[38958]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:11.826 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:11 smithi064 ceph-mon[38958]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-10T05:52:11.826 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:11 smithi064 ceph-mon[38958]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T05:52:11.826 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:11 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:52:11.826 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:11 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:52:11.826 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:11 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:52:12.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:11 smithi176 ceph-mon[30294]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:12.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:11 smithi176 ceph-mon[30294]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-10T05:52:12.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:11 smithi176 ceph-mon[30294]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T05:52:12.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:11 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:52:12.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:11 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:52:12.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:11 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:52:13.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:12 smithi064 ceph-mon[25185]: purged_snaps scrub starts 2024-02-10T05:52:13.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:12 smithi064 ceph-mon[25185]: purged_snaps scrub ok 2024-02-10T05:52:13.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:12 smithi064 ceph-mon[25185]: osd.1 [v2:172.21.15.64:6810/614169574,v1:172.21.15.64:6811/614169574] boot 2024-02-10T05:52:13.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:12 smithi064 ceph-mon[25185]: osdmap e13: 2 total, 2 up, 2 in 2024-02-10T05:52:13.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:12 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:52:13.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:12 smithi064 ceph-mon[38958]: purged_snaps scrub starts 2024-02-10T05:52:13.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:12 smithi064 ceph-mon[38958]: purged_snaps scrub ok 2024-02-10T05:52:13.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:12 smithi064 ceph-mon[38958]: osd.1 [v2:172.21.15.64:6810/614169574,v1:172.21.15.64:6811/614169574] boot 2024-02-10T05:52:13.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:12 smithi064 ceph-mon[38958]: osdmap e13: 2 total, 2 up, 2 in 2024-02-10T05:52:13.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:12 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:52:13.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:12 smithi176 ceph-mon[30294]: purged_snaps scrub starts 2024-02-10T05:52:13.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:12 smithi176 ceph-mon[30294]: purged_snaps scrub ok 2024-02-10T05:52:13.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:12 smithi176 ceph-mon[30294]: osd.1 [v2:172.21.15.64:6810/614169574,v1:172.21.15.64:6811/614169574] boot 2024-02-10T05:52:13.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:12 smithi176 ceph-mon[30294]: osdmap e13: 2 total, 2 up, 2 in 2024-02-10T05:52:13.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:12 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:52:13.847 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/nvme2n1 2024-02-10T05:52:13.847 INFO:teuthology.orchestra.run.smithi064.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-02-10T05:52:13.847 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme2n1 bs=1M count=10 conv=fsync 2024-02-10T05:52:13.847 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-02-10T05:52:13.848 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-02-10T05:52:13.848 INFO:teuthology.orchestra.run.smithi064.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125059 s, 838 MB/s 2024-02-10T05:52:13.848 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-02-10T05:52:14.122 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:13 smithi064 ceph-mon[25185]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:14.122 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:13 smithi064 ceph-mon[25185]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T05:52:14.123 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:13 smithi064 ceph-mon[38958]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:14.123 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:13 smithi064 ceph-mon[38958]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T05:52:14.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:13 smithi176 ceph-mon[30294]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:52:14.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:13 smithi176 ceph-mon[30294]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T05:52:14.769 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi064:/dev/nvme2n1 2024-02-10T05:52:15.010 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:52:15.420 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:15 smithi064 ceph-mon[25185]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:15.420 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:15 smithi064 ceph-mon[25185]: Detected new or changed devices on smithi064 2024-02-10T05:52:15.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:15 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:15.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:15 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:15.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:15 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:52:15.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:15 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:52:15.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:15 smithi064 ceph-mon[25185]: Adjusting osd_memory_target on smithi064 to 8032M 2024-02-10T05:52:15.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:15 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:15.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:15 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:15.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:15 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:52:15.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:15 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:15.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:15 smithi064 ceph-mon[38958]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:15.421 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:15 smithi064 ceph-mon[38958]: Detected new or changed devices on smithi064 2024-02-10T05:52:15.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:15 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:15.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:15 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:15.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:15 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:52:15.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:15 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:52:15.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:15 smithi064 ceph-mon[38958]: Adjusting osd_memory_target on smithi064 to 8032M 2024-02-10T05:52:15.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:15 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:15.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:15 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:15.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:15 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:52:15.422 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:15 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:15.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:15 smithi176 ceph-mon[30294]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:15.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:15 smithi176 ceph-mon[30294]: Detected new or changed devices on smithi064 2024-02-10T05:52:15.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:15 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:15.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:15 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:15.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:15 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:52:15.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:15 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:52:15.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:15 smithi176 ceph-mon[30294]: Adjusting osd_memory_target on smithi064 to 8032M 2024-02-10T05:52:15.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:15 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:15.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:15 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:15.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:15 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:52:15.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:15 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:17.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:17 smithi064 ceph-mon[25185]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:17.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:17 smithi064 ceph-mon[25185]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:17.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:17 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:52:17.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:17 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:52:17.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:17 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:17.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:17 smithi064 ceph-mon[38958]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:17.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:17 smithi064 ceph-mon[38958]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:17.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:17 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:52:17.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:17 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:52:17.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:17 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:17.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:17 smithi176 ceph-mon[30294]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:17.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:17 smithi176 ceph-mon[30294]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:17.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:17 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:52:17.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:17 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:52:17.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:17 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:19.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:19 smithi064 ceph-mon[25185]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:19.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:19 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/4270630498' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb433070-a584-427e-bb43-bf20be79b123"}]: dispatch 2024-02-10T05:52:19.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:19 smithi064 ceph-mon[25185]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb433070-a584-427e-bb43-bf20be79b123"}]: dispatch 2024-02-10T05:52:19.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:19 smithi064 ceph-mon[25185]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb433070-a584-427e-bb43-bf20be79b123"}]': finished 2024-02-10T05:52:19.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:19 smithi064 ceph-mon[25185]: osdmap e15: 3 total, 2 up, 3 in 2024-02-10T05:52:19.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:19 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:52:19.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:19 smithi064 ceph-mon[38958]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:19.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:19 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/4270630498' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb433070-a584-427e-bb43-bf20be79b123"}]: dispatch 2024-02-10T05:52:19.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:19 smithi064 ceph-mon[38958]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb433070-a584-427e-bb43-bf20be79b123"}]: dispatch 2024-02-10T05:52:19.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:19 smithi064 ceph-mon[38958]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb433070-a584-427e-bb43-bf20be79b123"}]': finished 2024-02-10T05:52:19.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:19 smithi064 ceph-mon[38958]: osdmap e15: 3 total, 2 up, 3 in 2024-02-10T05:52:19.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:19 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:52:19.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:19 smithi176 ceph-mon[30294]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:19.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:19 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/4270630498' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb433070-a584-427e-bb43-bf20be79b123"}]: dispatch 2024-02-10T05:52:19.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:19 smithi176 ceph-mon[30294]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb433070-a584-427e-bb43-bf20be79b123"}]: dispatch 2024-02-10T05:52:19.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:19 smithi176 ceph-mon[30294]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb433070-a584-427e-bb43-bf20be79b123"}]': finished 2024-02-10T05:52:19.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:19 smithi176 ceph-mon[30294]: osdmap e15: 3 total, 2 up, 3 in 2024-02-10T05:52:19.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:19 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:52:21.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:21 smithi064 ceph-mon[25185]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:21.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:21 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/3932864056' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:52:21.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:21 smithi064 ceph-mon[38958]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:21.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:21 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/3932864056' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:52:21.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:21 smithi176 ceph-mon[30294]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:21.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:21 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/3932864056' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:52:23.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:23 smithi064 ceph-mon[25185]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:23.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:23 smithi064 ceph-mon[38958]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:23.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:23 smithi176 ceph-mon[30294]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:25.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:25 smithi064 ceph-mon[25185]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:25.626 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:25 smithi064 ceph-mon[38958]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:25.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:25 smithi176 ceph-mon[30294]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:26.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:26 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T05:52:26.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:26 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:26.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:26 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T05:52:26.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:26 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:26.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:26 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T05:52:26.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:26 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:27.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:27 smithi064 ceph-mon[25185]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:27.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:27 smithi064 ceph-mon[25185]: Deploying daemon osd.2 on smithi064 2024-02-10T05:52:27.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:27 smithi064 ceph-mon[38958]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:27.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:27 smithi064 ceph-mon[38958]: Deploying daemon osd.2 on smithi064 2024-02-10T05:52:27.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:27 smithi176 ceph-mon[30294]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:27.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:27 smithi176 ceph-mon[30294]: Deploying daemon osd.2 on smithi064 2024-02-10T05:52:29.861 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:29 smithi064 ceph-mon[25185]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:29.861 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:29 smithi064 ceph-mon[38958]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:29.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:29 smithi176 ceph-mon[30294]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:31.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:31 smithi064 ceph-mon[25185]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:31.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:31 smithi064 ceph-mon[38958]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:31.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:31 smithi176 ceph-mon[30294]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:32.851 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:32 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:32.851 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:32 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:32.851 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:32 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:32.851 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:32 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:32.851 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:32 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:32.852 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:32 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:32.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:32 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:32.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:32 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:32.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:32 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:33.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:33 smithi064 ceph-mon[25185]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:33.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:33 smithi064 ceph-mon[38958]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:34.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:33 smithi176 ceph-mon[30294]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:35.708 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 2 on host 'smithi064' 2024-02-10T05:52:36.025 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:35 smithi064 ceph-mon[25185]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:36.026 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:35 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:36.026 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:35 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:36.026 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:35 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:36.026 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:35 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:52:36.026 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:35 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:36.026 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:35 smithi064 ceph-mon[25185]: from='osd.2 [v2:172.21.15.64:6818/2362209772,v1:172.21.15.64:6819/2362209772]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T05:52:36.026 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:35 smithi064 ceph-mon[25185]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T05:52:36.026 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:35 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:36.026 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:35 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:36.027 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:35 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:36.027 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:35 smithi064 ceph-mon[38958]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:36.027 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:35 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:36.027 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:35 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:36.027 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:35 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:36.027 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:35 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:52:36.027 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:35 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:36.027 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:35 smithi064 ceph-mon[38958]: from='osd.2 [v2:172.21.15.64:6818/2362209772,v1:172.21.15.64:6819/2362209772]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T05:52:36.027 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:35 smithi064 ceph-mon[38958]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T05:52:36.027 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:35 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:36.027 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:35 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:36.027 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:35 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:36.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:35 smithi176 ceph-mon[30294]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:36.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:35 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:36.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:35 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:36.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:35 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:36.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:35 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:52:36.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:35 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:36.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:35 smithi176 ceph-mon[30294]: from='osd.2 [v2:172.21.15.64:6818/2362209772,v1:172.21.15.64:6819/2362209772]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T05:52:36.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:35 smithi176 ceph-mon[30294]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T05:52:36.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:35 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:36.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:35 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:36.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:35 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:36.333 DEBUG:teuthology.orchestra.run.smithi064:osd.2> sudo journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.2.service 2024-02-10T05:52:36.335 INFO:tasks.cephadm:Deploying osd.3 on smithi064 with /dev/nvme1n1... 2024-02-10T05:52:36.335 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/nvme1n1 2024-02-10T05:52:37.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:36 smithi064 ceph-mon[38958]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T05:52:37.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:36 smithi064 ceph-mon[38958]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T05:52:37.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:36 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:52:37.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:36 smithi064 ceph-mon[38958]: from='osd.2 [v2:172.21.15.64:6818/2362209772,v1:172.21.15.64:6819/2362209772]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T05:52:37.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:36 smithi064 ceph-mon[38958]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T05:52:37.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:36 smithi064 ceph-mon[25185]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T05:52:37.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:36 smithi064 ceph-mon[25185]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T05:52:37.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:36 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:52:37.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:36 smithi064 ceph-mon[25185]: from='osd.2 [v2:172.21.15.64:6818/2362209772,v1:172.21.15.64:6819/2362209772]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T05:52:37.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:36 smithi064 ceph-mon[25185]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T05:52:37.127 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 05:52:36 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-2[58170]: 2024-02-10T05:52:36.879+0000 7fc48cc22700 -1 osd.2 0 waiting for initial osdmap 2024-02-10T05:52:37.127 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 05:52:36 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-2[58170]: 2024-02-10T05:52:36.883+0000 7fc488a15700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:52:37.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:36 smithi176 ceph-mon[30294]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T05:52:37.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:36 smithi176 ceph-mon[30294]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T05:52:37.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:36 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:52:37.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:36 smithi176 ceph-mon[30294]: from='osd.2 [v2:172.21.15.64:6818/2362209772,v1:172.21.15.64:6819/2362209772]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T05:52:37.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:36 smithi176 ceph-mon[30294]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T05:52:37.504 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:52:37.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:37 smithi064 ceph-mon[25185]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:37.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:37 smithi064 ceph-mon[25185]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-10T05:52:37.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:37 smithi064 ceph-mon[25185]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T05:52:37.985 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:37 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:52:37.986 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:37 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:52:37.986 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:37 smithi064 ceph-mon[38958]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:37.986 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:37 smithi064 ceph-mon[38958]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-10T05:52:37.986 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:37 smithi064 ceph-mon[38958]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T05:52:37.986 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:37 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:52:37.986 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:37 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:52:38.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:37 smithi176 ceph-mon[30294]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:38.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:37 smithi176 ceph-mon[30294]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-10T05:52:38.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:37 smithi176 ceph-mon[30294]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T05:52:38.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:37 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:52:38.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:37 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:52:39.119 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:38 smithi064 ceph-mon[25185]: purged_snaps scrub starts 2024-02-10T05:52:39.119 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:38 smithi064 ceph-mon[25185]: purged_snaps scrub ok 2024-02-10T05:52:39.119 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:38 smithi064 ceph-mon[25185]: osd.2 [v2:172.21.15.64:6818/2362209772,v1:172.21.15.64:6819/2362209772] boot 2024-02-10T05:52:39.120 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:38 smithi064 ceph-mon[25185]: osdmap e18: 3 total, 3 up, 3 in 2024-02-10T05:52:39.120 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:38 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:52:39.120 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:38 smithi064 ceph-mon[25185]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:39.120 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:38 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-10T05:52:39.120 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:38 smithi064 ceph-mon[38958]: purged_snaps scrub starts 2024-02-10T05:52:39.120 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:38 smithi064 ceph-mon[38958]: purged_snaps scrub ok 2024-02-10T05:52:39.120 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:38 smithi064 ceph-mon[38958]: osd.2 [v2:172.21.15.64:6818/2362209772,v1:172.21.15.64:6819/2362209772] boot 2024-02-10T05:52:39.120 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:38 smithi064 ceph-mon[38958]: osdmap e18: 3 total, 3 up, 3 in 2024-02-10T05:52:39.120 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:38 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:52:39.121 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:38 smithi064 ceph-mon[38958]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:39.121 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:38 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-10T05:52:39.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:38 smithi176 ceph-mon[30294]: purged_snaps scrub starts 2024-02-10T05:52:39.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:38 smithi176 ceph-mon[30294]: purged_snaps scrub ok 2024-02-10T05:52:39.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:38 smithi176 ceph-mon[30294]: osd.2 [v2:172.21.15.64:6818/2362209772,v1:172.21.15.64:6819/2362209772] boot 2024-02-10T05:52:39.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:38 smithi176 ceph-mon[30294]: osdmap e18: 3 total, 3 up, 3 in 2024-02-10T05:52:39.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:38 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:52:39.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:38 smithi176 ceph-mon[30294]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:52:39.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:38 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-10T05:52:40.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:39 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-10T05:52:40.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:39 smithi176 ceph-mon[30294]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T05:52:40.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:39 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T05:52:40.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:39 smithi176 ceph-mon[30294]: Detected new or changed devices on smithi064 2024-02-10T05:52:40.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:39 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:40.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:39 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:40.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:39 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:52:40.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:39 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:52:40.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:39 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:52:40.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:39 smithi176 ceph-mon[30294]: Adjusting osd_memory_target on smithi064 to 5354M 2024-02-10T05:52:40.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:39 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:40.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:39 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:40.240 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:39 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:52:40.240 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:39 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:40.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-10T05:52:40.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[25185]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T05:52:40.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T05:52:40.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[25185]: Detected new or changed devices on smithi064 2024-02-10T05:52:40.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:40.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:40.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:52:40.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:52:40.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:52:40.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[25185]: Adjusting osd_memory_target on smithi064 to 5354M 2024-02-10T05:52:40.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:40.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:40.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:52:40.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:40.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-10T05:52:40.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[38958]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T05:52:40.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T05:52:40.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[38958]: Detected new or changed devices on smithi064 2024-02-10T05:52:40.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:40.378 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:40.378 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:52:40.378 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:52:40.378 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:52:40.378 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[38958]: Adjusting osd_memory_target on smithi064 to 5354M 2024-02-10T05:52:40.378 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:40.378 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:40.378 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:52:40.378 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:39 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:40.437 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/nvme1n1 2024-02-10T05:52:40.437 INFO:teuthology.orchestra.run.smithi064.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-02-10T05:52:40.437 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme1n1 bs=1M count=10 conv=fsync 2024-02-10T05:52:40.437 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-02-10T05:52:40.437 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-02-10T05:52:40.437 INFO:teuthology.orchestra.run.smithi064.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012674 s, 827 MB/s 2024-02-10T05:52:40.437 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-02-10T05:52:40.626 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:52:40 smithi064 sudo[64207]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme4n1 2024-02-10T05:52:40.626 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:52:40 smithi064 sudo[64207]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:52:40.626 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:52:40 smithi064 sudo[64207]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:52:40.626 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:52:40 smithi064 sudo[64207]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:52:40.903 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:52:40 smithi064 sudo[64231]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-02-10T05:52:40.904 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:52:40 smithi064 sudo[64231]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:52:40.904 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:52:40 smithi064 sudo[64231]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:52:40.904 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:52:40 smithi064 sudo[64231]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:52:41.024 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi064:/dev/nvme1n1 2024-02-10T05:52:41.204 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 05:52:40 smithi064 sudo[64234]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme3n1 2024-02-10T05:52:41.204 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 05:52:40 smithi064 sudo[64234]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:52:41.204 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 05:52:40 smithi064 sudo[64234]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:52:41.204 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 05:52:40 smithi064 sudo[64234]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:52:41.205 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:40 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T05:52:41.205 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:40 smithi064 ceph-mon[25185]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T05:52:41.205 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:40 smithi064 ceph-mon[25185]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:52:41.205 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:40 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:41.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:40 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T05:52:41.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:40 smithi064 ceph-mon[38958]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T05:52:41.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:40 smithi064 ceph-mon[38958]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:52:41.205 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:40 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:41.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:40 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T05:52:41.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:40 smithi176 ceph-mon[30294]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T05:52:41.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:40 smithi176 ceph-mon[30294]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:52:41.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:40 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:41.287 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:52:41.494 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 05:52:41 smithi064 sudo[64263]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme linux smart-log-add --json /dev/nvme3n1 2024-02-10T05:52:41.494 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 05:52:41 smithi064 sudo[64263]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:52:41.495 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 05:52:41 smithi064 sudo[64263]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:52:41.495 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 05:52:41 smithi064 sudo[64263]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:52:41.495 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 05:52:41 smithi064 sudo[64319]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme2n1 2024-02-10T05:52:41.766 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 05:52:41 smithi064 sudo[64319]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:52:41.766 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 05:52:41 smithi064 sudo[64319]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:52:41.766 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 05:52:41 smithi064 sudo[64319]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:52:42.044 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 05:52:41 smithi064 sudo[64340]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme linux smart-log-add --json /dev/nvme2n1 2024-02-10T05:52:42.044 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 05:52:41 smithi064 sudo[64340]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:52:42.044 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 05:52:41 smithi064 sudo[64340]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:52:42.044 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 05:52:41 smithi064 sudo[64340]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:52:42.308 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:42 smithi064 sudo[64346]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T05:52:42.308 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:42 smithi064 sudo[64346]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:52:42.308 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:42 smithi064 sudo[64346]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:52:42.308 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:42 smithi064 sudo[64346]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:52:42.577 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:42 smithi176 ceph-mon[30294]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T05:52:42.577 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:42 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:42.577 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:42 smithi176 ceph-mon[30294]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-10T05:52:42.577 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:42 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:42.577 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:42 smithi176 ceph-mon[30294]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:52:42.597 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:42 smithi064 ceph-mon[38958]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T05:52:42.597 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:42 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:42.598 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:42 smithi064 ceph-mon[38958]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-10T05:52:42.598 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:42 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:42.598 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:42 smithi064 ceph-mon[38958]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:52:42.598 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:42 smithi064 sudo[64351]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T05:52:42.598 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:42 smithi064 sudo[64351]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:52:42.598 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:42 smithi064 sudo[64351]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:52:42.598 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:42 smithi064 sudo[64351]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:52:42.598 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:42 smithi064 ceph-mon[25185]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T05:52:42.598 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:42 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:42.598 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:42 smithi064 ceph-mon[25185]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-10T05:52:42.598 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:42 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:42.599 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:42 smithi064 ceph-mon[25185]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:52:42.829 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:42 smithi176 sudo[32868]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T05:52:42.830 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:42 smithi176 sudo[32868]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:52:42.830 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:42 smithi176 sudo[32868]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:52:42.830 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:42 smithi176 sudo[32868]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:52:43.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:42 smithi176 sudo[32871]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T05:52:43.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:42 smithi176 sudo[32871]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:52:43.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:42 smithi176 sudo[32871]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:52:43.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:42 smithi176 sudo[32871]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:52:43.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 sudo[64468]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T05:52:43.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 sudo[64468]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:52:43.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 sudo[64468]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:52:43.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 sudo[64468]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:52:43.647 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[25185]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:52:43.647 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[25185]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:52:43.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:52:43.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:52:43.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:52:43.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:43.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[25185]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:52:43.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:52:43.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:52:43.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:52:43.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[25185]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:52:43.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:43.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:52:43.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:52:43.648 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:52:43.649 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:52:43.649 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:52:43.649 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:43.649 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 sudo[64471]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T05:52:43.649 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 sudo[64471]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:52:43.649 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 sudo[64471]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:52:43.649 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 sudo[64471]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:52:43.649 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[38958]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:52:43.649 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[38958]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:52:43.649 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:52:43.649 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:52:43.649 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:52:43.650 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:43.650 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[38958]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:52:43.650 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:52:43.650 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:52:43.650 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:52:43.650 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[38958]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:52:43.650 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:43.650 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:52:43.650 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:52:43.650 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:52:43.650 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:52:43.650 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:52:43.650 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:43 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:43.738 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:43 smithi176 ceph-mon[30294]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:52:43.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:43 smithi176 ceph-mon[30294]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:52:43.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:43 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:52:43.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:43 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:52:43.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:43 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:52:43.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:43 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:43.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:43 smithi176 ceph-mon[30294]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:52:43.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:43 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:52:43.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:43 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:52:43.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:43 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:52:43.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:43 smithi176 ceph-mon[30294]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:52:43.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:43 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:43.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:43 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:52:43.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:43 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:52:43.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:43 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:52:43.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:43 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:52:43.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:43 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:52:43.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:43 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:44.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:44 smithi064 ceph-mon[25185]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:52:44.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:44 smithi064 ceph-mon[25185]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:44.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:44 smithi064 ceph-mon[25185]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:52:44.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:44 smithi064 ceph-mon[38958]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:52:44.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:44 smithi064 ceph-mon[38958]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:44.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:44 smithi064 ceph-mon[38958]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:52:44.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:44 smithi176 ceph-mon[30294]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:52:44.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:44 smithi176 ceph-mon[30294]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:52:44.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:44 smithi176 ceph-mon[30294]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:52:45.738 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:45 smithi176 ceph-mon[30294]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:52:45.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:45 smithi064 ceph-mon[25185]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:52:45.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:45 smithi064 ceph-mon[38958]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:52:46.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:46 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/177903506' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb31284d-6c1a-4f79-928a-bb758b45e196"}]: dispatch 2024-02-10T05:52:46.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:46 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/177903506' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb31284d-6c1a-4f79-928a-bb758b45e196"}]': finished 2024-02-10T05:52:46.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:46 smithi064 ceph-mon[38958]: osdmap e22: 4 total, 3 up, 4 in 2024-02-10T05:52:46.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:46 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:52:46.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:46 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/177903506' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb31284d-6c1a-4f79-928a-bb758b45e196"}]: dispatch 2024-02-10T05:52:46.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:46 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/177903506' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb31284d-6c1a-4f79-928a-bb758b45e196"}]': finished 2024-02-10T05:52:46.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:46 smithi064 ceph-mon[25185]: osdmap e22: 4 total, 3 up, 4 in 2024-02-10T05:52:46.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:46 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:52:46.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:46 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/177903506' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bb31284d-6c1a-4f79-928a-bb758b45e196"}]: dispatch 2024-02-10T05:52:46.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:46 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/177903506' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bb31284d-6c1a-4f79-928a-bb758b45e196"}]': finished 2024-02-10T05:52:46.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:46 smithi176 ceph-mon[30294]: osdmap e22: 4 total, 3 up, 4 in 2024-02-10T05:52:46.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:46 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:52:47.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:47 smithi176 ceph-mon[30294]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:52:47.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:47 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/954716715' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:52:48.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:47 smithi064 ceph-mon[25185]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:52:48.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:47 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/954716715' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:52:48.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:47 smithi064 ceph-mon[38958]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:52:48.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:47 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/954716715' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:52:49.738 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:49 smithi064 ceph-mon[25185]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-02-10T05:52:49.739 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:49 smithi064 ceph-mon[38958]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-02-10T05:52:49.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:49 smithi176 ceph-mon[30294]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-02-10T05:52:51.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:51 smithi064 ceph-mon[25185]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-02-10T05:52:51.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:51 smithi064 ceph-mon[38958]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-02-10T05:52:51.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:51 smithi176 ceph-mon[30294]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-02-10T05:52:53.874 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:53 smithi064 ceph-mon[25185]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-02-10T05:52:53.874 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:53 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T05:52:53.874 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:53 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:53.874 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:53 smithi064 ceph-mon[25185]: Deploying daemon osd.3 on smithi064 2024-02-10T05:52:53.875 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:53 smithi064 ceph-mon[38958]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-02-10T05:52:53.875 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:53 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T05:52:53.875 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:53 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:53.875 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:53 smithi064 ceph-mon[38958]: Deploying daemon osd.3 on smithi064 2024-02-10T05:52:53.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:53 smithi176 ceph-mon[30294]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-02-10T05:52:53.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:53 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-10T05:52:53.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:53 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:52:53.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:53 smithi176 ceph-mon[30294]: Deploying daemon osd.3 on smithi064 2024-02-10T05:52:55.752 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:55 smithi064 ceph-mon[25185]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-10T05:52:55.753 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:55 smithi064 ceph-mon[38958]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-10T05:52:55.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:55 smithi176 ceph-mon[30294]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-10T05:52:57.851 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:57 smithi064 ceph-mon[25185]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 76 KiB/s wr, 5 op/s 2024-02-10T05:52:57.852 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:57 smithi064 ceph-mon[38958]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 76 KiB/s wr, 5 op/s 2024-02-10T05:52:57.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:57 smithi176 ceph-mon[30294]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 76 KiB/s wr, 5 op/s 2024-02-10T05:52:59.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:59 smithi064 ceph-mon[25185]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-10T05:52:59.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:59 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:59.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:59 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:59.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:52:59 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:59.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:59 smithi064 ceph-mon[38958]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-10T05:52:59.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:59 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:59.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:59 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:59.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:52:59 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:52:59.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:59 smithi176 ceph-mon[30294]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-10T05:52:59.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:59 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:59.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:59 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:52:59.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:52:59 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:01.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:01 smithi064 ceph-mon[38958]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:53:01.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:01 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:01.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:01 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:01.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:01 smithi064 ceph-mon[38958]: from='osd.3 [v2:172.21.15.64:6826/256208898,v1:172.21.15.64:6827/256208898]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T05:53:01.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:01 smithi064 ceph-mon[38958]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T05:53:01.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:01 smithi064 ceph-mon[25185]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:53:01.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:01 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:01.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:01 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:01.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:01 smithi064 ceph-mon[25185]: from='osd.3 [v2:172.21.15.64:6826/256208898,v1:172.21.15.64:6827/256208898]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T05:53:01.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:01 smithi064 ceph-mon[25185]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T05:53:01.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:01 smithi176 ceph-mon[30294]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:53:01.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:01 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:01.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:01 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:01.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:01 smithi176 ceph-mon[30294]: from='osd.3 [v2:172.21.15.64:6826/256208898,v1:172.21.15.64:6827/256208898]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T05:53:01.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:01 smithi176 ceph-mon[30294]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-10T05:53:02.092 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 3 on host 'smithi064' 2024-02-10T05:53:02.728 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:02 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:02.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:02 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:02.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:02 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:02.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:02 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:02.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:02 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:02.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:02 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:02.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:02 smithi064 ceph-mon[25185]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T05:53:02.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:02 smithi064 ceph-mon[25185]: osdmap e23: 4 total, 3 up, 4 in 2024-02-10T05:53:02.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:02 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:53:02.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:02 smithi064 ceph-mon[25185]: from='osd.3 [v2:172.21.15.64:6826/256208898,v1:172.21.15.64:6827/256208898]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T05:53:02.729 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:02 smithi064 ceph-mon[25185]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T05:53:02.730 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:02 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:02.730 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:02 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:02.730 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:02 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:02.730 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:02 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:02.730 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:02 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:02.730 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:02 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:02.730 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:02 smithi064 ceph-mon[38958]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T05:53:02.730 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:02 smithi064 ceph-mon[38958]: osdmap e23: 4 total, 3 up, 4 in 2024-02-10T05:53:02.730 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:02 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:53:02.730 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:02 smithi064 ceph-mon[38958]: from='osd.3 [v2:172.21.15.64:6826/256208898,v1:172.21.15.64:6827/256208898]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T05:53:02.730 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:02 smithi064 ceph-mon[38958]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T05:53:02.760 DEBUG:teuthology.orchestra.run.smithi064:osd.3> sudo journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.3.service 2024-02-10T05:53:02.762 INFO:tasks.cephadm:Deploying osd.4 on smithi176 with /dev/nvme4n1... 2024-02-10T05:53:02.762 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-02-10T05:53:02.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:02 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:02.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:02 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:02.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:02 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:02.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:02 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:02.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:02 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:02.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:02 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:02.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:02 smithi176 ceph-mon[30294]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-10T05:53:02.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:02 smithi176 ceph-mon[30294]: osdmap e23: 4 total, 3 up, 4 in 2024-02-10T05:53:02.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:02 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:53:02.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:02 smithi176 ceph-mon[30294]: from='osd.3 [v2:172.21.15.64:6826/256208898,v1:172.21.15.64:6827/256208898]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T05:53:02.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:02 smithi176 ceph-mon[30294]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-10T05:53:03.007 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:53:03.945 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:03 smithi176 ceph-mon[30294]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:53:03.945 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:03 smithi176 ceph-mon[30294]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-10T05:53:03.945 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:03 smithi176 ceph-mon[30294]: osdmap e24: 4 total, 3 up, 4 in 2024-02-10T05:53:03.945 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:03 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:53:03.945 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:03 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:53:03.957 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:03 smithi064 ceph-mon[25185]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:53:03.957 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:03 smithi064 ceph-mon[25185]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-10T05:53:03.957 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:03 smithi064 ceph-mon[25185]: osdmap e24: 4 total, 3 up, 4 in 2024-02-10T05:53:03.957 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:03 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:53:03.957 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:03 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:53:03.957 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:03 smithi064 ceph-mon[38958]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:53:03.958 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:03 smithi064 ceph-mon[38958]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-10T05:53:03.958 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:03 smithi064 ceph-mon[38958]: osdmap e24: 4 total, 3 up, 4 in 2024-02-10T05:53:03.958 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:03 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:53:03.958 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:03 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:53:03.958 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 05:53:03 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-3[66346]: 2024-02-10T05:53:03.768+0000 7f0f54839700 -1 osd.3 0 waiting for initial osdmap 2024-02-10T05:53:03.958 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 05:53:03 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-3[66346]: 2024-02-10T05:53:03.772+0000 7f0f4c624700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:53:04.751 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping: /dev/nvme4n1 2024-02-10T05:53:04.751 INFO:teuthology.orchestra.run.smithi176.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-02-10T05:53:04.751 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-02-10T05:53:04.751 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10+0 records in 2024-02-10T05:53:04.751 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2024-02-10T05:53:04.751 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127596 s, 822 MB/s 2024-02-10T05:53:04.752 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping successful for: 2024-02-10T05:53:05.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:04 smithi064 ceph-mon[25185]: purged_snaps scrub starts 2024-02-10T05:53:05.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:04 smithi064 ceph-mon[25185]: purged_snaps scrub ok 2024-02-10T05:53:05.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:04 smithi064 ceph-mon[25185]: from='osd.3 ' entity='osd.3' 2024-02-10T05:53:05.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:04 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:53:05.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:04 smithi064 ceph-mon[38958]: purged_snaps scrub starts 2024-02-10T05:53:05.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:04 smithi064 ceph-mon[38958]: purged_snaps scrub ok 2024-02-10T05:53:05.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:04 smithi064 ceph-mon[38958]: from='osd.3 ' entity='osd.3' 2024-02-10T05:53:05.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:04 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:53:05.132 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:04 smithi176 ceph-mon[30294]: purged_snaps scrub starts 2024-02-10T05:53:05.132 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:04 smithi176 ceph-mon[30294]: purged_snaps scrub ok 2024-02-10T05:53:05.132 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:04 smithi176 ceph-mon[30294]: from='osd.3 ' entity='osd.3' 2024-02-10T05:53:05.133 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:04 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:53:05.368 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi176:/dev/nvme4n1 2024-02-10T05:53:05.618 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:53:05.866 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:05 smithi064 ceph-mon[25185]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:53:05.866 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:05 smithi064 ceph-mon[25185]: osd.3 [v2:172.21.15.64:6826/256208898,v1:172.21.15.64:6827/256208898] boot 2024-02-10T05:53:05.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:05 smithi064 ceph-mon[25185]: osdmap e25: 4 total, 4 up, 4 in 2024-02-10T05:53:05.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:05 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:53:05.867 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:05 smithi064 ceph-mon[38958]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:53:05.867 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:05 smithi064 ceph-mon[38958]: osd.3 [v2:172.21.15.64:6826/256208898,v1:172.21.15.64:6827/256208898] boot 2024-02-10T05:53:05.867 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:05 smithi064 ceph-mon[38958]: osdmap e25: 4 total, 4 up, 4 in 2024-02-10T05:53:05.867 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:05 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:53:05.996 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:05 smithi176 ceph-mon[30294]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:53:05.996 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:05 smithi176 ceph-mon[30294]: osd.3 [v2:172.21.15.64:6826/256208898,v1:172.21.15.64:6827/256208898] boot 2024-02-10T05:53:05.996 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:05 smithi176 ceph-mon[30294]: osdmap e25: 4 total, 4 up, 4 in 2024-02-10T05:53:05.996 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:05 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:53:06.893 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:06 smithi176 ceph-mon[30294]: osdmap e26: 4 total, 4 up, 4 in 2024-02-10T05:53:06.893 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:06 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:06.893 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:06 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:06.893 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:06 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:06.893 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:06 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:06.893 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:06 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:06.893 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:06 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:06.893 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:06 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:06.894 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:06 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:06.894 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:06 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:06.894 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:06 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:07.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:06 smithi064 ceph-mon[25185]: osdmap e26: 4 total, 4 up, 4 in 2024-02-10T05:53:07.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:06 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:07.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:06 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:07.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:06 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:07.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:06 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:07.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:06 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:07.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:06 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:07.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:06 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:07.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:06 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:07.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:06 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:07.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:06 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:07.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:06 smithi064 ceph-mon[38958]: osdmap e26: 4 total, 4 up, 4 in 2024-02-10T05:53:07.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:07.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:07.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:07.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:07.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:07.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:07.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:07.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:07.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:07.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:08.039 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:07 smithi176 ceph-mon[30294]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:08.039 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:07 smithi176 ceph-mon[30294]: Detected new or changed devices on smithi064 2024-02-10T05:53:08.039 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:07 smithi176 ceph-mon[30294]: Adjusting osd_memory_target on smithi064 to 4016M 2024-02-10T05:53:08.039 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:07 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:53:08.039 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:07 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:53:08.039 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:07 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:08.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:07 smithi064 ceph-mon[25185]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:08.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:07 smithi064 ceph-mon[25185]: Detected new or changed devices on smithi064 2024-02-10T05:53:08.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:07 smithi064 ceph-mon[25185]: Adjusting osd_memory_target on smithi064 to 4016M 2024-02-10T05:53:08.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:07 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:53:08.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:07 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:53:08.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:07 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:08.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:07 smithi064 ceph-mon[38958]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:08.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:07 smithi064 ceph-mon[38958]: Detected new or changed devices on smithi064 2024-02-10T05:53:08.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:07 smithi064 ceph-mon[38958]: Adjusting osd_memory_target on smithi064 to 4016M 2024-02-10T05:53:08.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:07 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:53:08.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:07 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:53:08.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:07 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:08.950 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:08 smithi176 ceph-mon[30294]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:09.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:08 smithi064 ceph-mon[25185]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:09.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:08 smithi064 ceph-mon[38958]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:10.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:09 smithi064 ceph-mon[25185]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:10.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:09 smithi064 ceph-mon[38958]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:10.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:09 smithi176 ceph-mon[30294]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:11.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:10 smithi064 ceph-mon[25185]: from='client.? 172.21.15.176:0/1290322199' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a5c7c64-fbb4-4d8b-b3d5-144164369304"}]: dispatch 2024-02-10T05:53:11.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:10 smithi064 ceph-mon[25185]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a5c7c64-fbb4-4d8b-b3d5-144164369304"}]: dispatch 2024-02-10T05:53:11.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:10 smithi064 ceph-mon[25185]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a5c7c64-fbb4-4d8b-b3d5-144164369304"}]': finished 2024-02-10T05:53:11.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:10 smithi064 ceph-mon[25185]: osdmap e27: 5 total, 4 up, 5 in 2024-02-10T05:53:11.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:10 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:53:11.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:10 smithi064 ceph-mon[38958]: from='client.? 172.21.15.176:0/1290322199' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a5c7c64-fbb4-4d8b-b3d5-144164369304"}]: dispatch 2024-02-10T05:53:11.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:10 smithi064 ceph-mon[38958]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a5c7c64-fbb4-4d8b-b3d5-144164369304"}]: dispatch 2024-02-10T05:53:11.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:10 smithi064 ceph-mon[38958]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a5c7c64-fbb4-4d8b-b3d5-144164369304"}]': finished 2024-02-10T05:53:11.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:10 smithi064 ceph-mon[38958]: osdmap e27: 5 total, 4 up, 5 in 2024-02-10T05:53:11.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:10 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:53:11.139 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:10 smithi176 ceph-mon[30294]: from='client.? 172.21.15.176:0/1290322199' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a5c7c64-fbb4-4d8b-b3d5-144164369304"}]: dispatch 2024-02-10T05:53:11.139 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:10 smithi176 ceph-mon[30294]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a5c7c64-fbb4-4d8b-b3d5-144164369304"}]: dispatch 2024-02-10T05:53:11.140 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:10 smithi176 ceph-mon[30294]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a5c7c64-fbb4-4d8b-b3d5-144164369304"}]': finished 2024-02-10T05:53:11.140 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:10 smithi176 ceph-mon[30294]: osdmap e27: 5 total, 4 up, 5 in 2024-02-10T05:53:11.140 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:10 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:53:12.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:11 smithi064 ceph-mon[25185]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:12.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:11 smithi064 ceph-mon[25185]: from='client.? 172.21.15.176:0/3565695652' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:53:12.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:11 smithi064 ceph-mon[38958]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:12.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:11 smithi064 ceph-mon[38958]: from='client.? 172.21.15.176:0/3565695652' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:53:12.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:11 smithi176 ceph-mon[30294]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:12.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:11 smithi176 ceph-mon[30294]: from='client.? 172.21.15.176:0/3565695652' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:53:13.943 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:13 smithi176 ceph-mon[30294]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:14.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:13 smithi064 ceph-mon[25185]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:14.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:13 smithi064 ceph-mon[38958]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:16.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:15 smithi064 ceph-mon[25185]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:16.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:15 smithi064 ceph-mon[38958]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:16.210 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:15 smithi176 ceph-mon[30294]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:16.991 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:16 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T05:53:16.991 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:16 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:17.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:16 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T05:53:17.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:16 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:17.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:16 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-10T05:53:17.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:16 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:17.928 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:17 smithi176 ceph-mon[30294]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:17.928 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:17 smithi176 ceph-mon[30294]: Deploying daemon osd.4 on smithi176 2024-02-10T05:53:18.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:17 smithi064 ceph-mon[25185]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:18.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:17 smithi064 ceph-mon[25185]: Deploying daemon osd.4 on smithi176 2024-02-10T05:53:18.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:17 smithi064 ceph-mon[38958]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:18.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:17 smithi064 ceph-mon[38958]: Deploying daemon osd.4 on smithi176 2024-02-10T05:53:19.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:19 smithi176 ceph-mon[30294]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:20.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:19 smithi064 ceph-mon[25185]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:20.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:19 smithi064 ceph-mon[38958]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:22.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:21 smithi064 ceph-mon[25185]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:22.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:21 smithi064 ceph-mon[38958]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:22.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:21 smithi176 ceph-mon[30294]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:23.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:23 smithi176 ceph-mon[30294]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:23.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:23 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:23.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:23 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:23.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:23 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:23.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:23 smithi064 ceph-mon[25185]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:23.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:23 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:23.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:23 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:23.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:23 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:23.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:23 smithi064 ceph-mon[38958]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:23.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:23 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:23.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:23 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:23.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:23 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:25.087 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:25 smithi176 ceph-mon[30294]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:25.134 INFO:teuthology.orchestra.run.smithi176.stdout:Created osd(s) 4 on host 'smithi176' 2024-02-10T05:53:25.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:25 smithi064 ceph-mon[25185]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:25.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:25 smithi064 ceph-mon[38958]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:25.698 DEBUG:teuthology.orchestra.run.smithi176:osd.4> sudo journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.4.service 2024-02-10T05:53:25.700 INFO:tasks.cephadm:Deploying osd.5 on smithi176 with /dev/nvme3n1... 2024-02-10T05:53:25.700 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/nvme3n1 2024-02-10T05:53:26.008 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:53:26.258 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:26 smithi176 ceph-mon[30294]: from='osd.4 [v2:172.21.15.176:6800/3371979738,v1:172.21.15.176:6801/3371979738]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:53:26.259 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:26 smithi176 ceph-mon[30294]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:53:26.259 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:26 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:26.259 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:26 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:26.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:26 smithi064 ceph-mon[25185]: from='osd.4 [v2:172.21.15.176:6800/3371979738,v1:172.21.15.176:6801/3371979738]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:53:26.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:26 smithi064 ceph-mon[25185]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:53:26.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:26 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:26.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:26 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:26.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:26 smithi064 ceph-mon[38958]: from='osd.4 [v2:172.21.15.176:6800/3371979738,v1:172.21.15.176:6801/3371979738]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:53:26.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:26 smithi064 ceph-mon[38958]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-10T05:53:26.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:26 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:26.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:26 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:27.307 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:27 smithi176 ceph-mon[30294]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:27.307 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:27 smithi176 ceph-mon[30294]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T05:53:27.307 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:27 smithi176 ceph-mon[30294]: osdmap e28: 5 total, 4 up, 5 in 2024-02-10T05:53:27.307 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:27 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:53:27.307 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:27 smithi176 ceph-mon[30294]: from='osd.4 [v2:172.21.15.176:6800/3371979738,v1:172.21.15.176:6801/3371979738]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-10T05:53:27.307 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:27 smithi176 ceph-mon[30294]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-10T05:53:27.307 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:27 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:27.307 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:27 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:27.307 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:53:27 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-4[35167]: 2024-02-10T05:53:27.022+0000 7f5feb59e700 -1 osd.4 0 waiting for initial osdmap 2024-02-10T05:53:27.308 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:53:27 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-4[35167]: 2024-02-10T05:53:27.026+0000 7f5fe7391700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:53:27.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:27 smithi064 ceph-mon[25185]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:27.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:27 smithi064 ceph-mon[25185]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T05:53:27.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:27 smithi064 ceph-mon[25185]: osdmap e28: 5 total, 4 up, 5 in 2024-02-10T05:53:27.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:27 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:53:27.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:27 smithi064 ceph-mon[25185]: from='osd.4 [v2:172.21.15.176:6800/3371979738,v1:172.21.15.176:6801/3371979738]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-10T05:53:27.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:27 smithi064 ceph-mon[25185]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-10T05:53:27.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:27 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:27.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:27 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:27.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:27 smithi064 ceph-mon[38958]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:27.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:27 smithi064 ceph-mon[38958]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-10T05:53:27.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:27 smithi064 ceph-mon[38958]: osdmap e28: 5 total, 4 up, 5 in 2024-02-10T05:53:27.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:27 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:53:27.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:27 smithi064 ceph-mon[38958]: from='osd.4 [v2:172.21.15.176:6800/3371979738,v1:172.21.15.176:6801/3371979738]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-10T05:53:27.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:27 smithi064 ceph-mon[38958]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-10T05:53:27.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:27 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:27.378 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:27 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:27.875 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping: /dev/nvme3n1 2024-02-10T05:53:27.876 INFO:teuthology.orchestra.run.smithi176.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-02-10T05:53:27.876 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme3n1 bs=1M count=10 conv=fsync 2024-02-10T05:53:27.876 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10+0 records in 2024-02-10T05:53:27.876 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2024-02-10T05:53:27.876 INFO:teuthology.orchestra.run.smithi176.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0153178 s, 685 MB/s 2024-02-10T05:53:27.876 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping successful for: 2024-02-10T05:53:28.130 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:28 smithi176 ceph-mon[30294]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-02-10T05:53:28.130 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:28 smithi176 ceph-mon[30294]: osdmap e29: 5 total, 4 up, 5 in 2024-02-10T05:53:28.130 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:28 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:53:28.130 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:28 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:53:28.130 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:28 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:53:28.296 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:28 smithi064 ceph-mon[25185]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-02-10T05:53:28.296 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:28 smithi064 ceph-mon[25185]: osdmap e29: 5 total, 4 up, 5 in 2024-02-10T05:53:28.296 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:28 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:53:28.296 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:28 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:53:28.296 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:28 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:53:28.296 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:28 smithi064 ceph-mon[38958]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-02-10T05:53:28.296 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:28 smithi064 ceph-mon[38958]: osdmap e29: 5 total, 4 up, 5 in 2024-02-10T05:53:28.297 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:28 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:53:28.297 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:28 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:53:28.297 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:28 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:53:28.657 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi176:/dev/nvme3n1 2024-02-10T05:53:29.124 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:53:29.325 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:29 smithi176 ceph-mon[30294]: purged_snaps scrub starts 2024-02-10T05:53:29.326 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:29 smithi176 ceph-mon[30294]: purged_snaps scrub ok 2024-02-10T05:53:29.326 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:29 smithi176 ceph-mon[30294]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:29.326 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:29 smithi176 ceph-mon[30294]: osd.4 [v2:172.21.15.176:6800/3371979738,v1:172.21.15.176:6801/3371979738] boot 2024-02-10T05:53:29.326 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:29 smithi176 ceph-mon[30294]: osdmap e30: 5 total, 5 up, 5 in 2024-02-10T05:53:29.326 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:29 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:53:29.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:29 smithi064 ceph-mon[25185]: purged_snaps scrub starts 2024-02-10T05:53:29.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:29 smithi064 ceph-mon[25185]: purged_snaps scrub ok 2024-02-10T05:53:29.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:29 smithi064 ceph-mon[25185]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:29.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:29 smithi064 ceph-mon[25185]: osd.4 [v2:172.21.15.176:6800/3371979738,v1:172.21.15.176:6801/3371979738] boot 2024-02-10T05:53:29.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:29 smithi064 ceph-mon[25185]: osdmap e30: 5 total, 5 up, 5 in 2024-02-10T05:53:29.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:29 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:53:29.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:29 smithi064 ceph-mon[38958]: purged_snaps scrub starts 2024-02-10T05:53:29.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:29 smithi064 ceph-mon[38958]: purged_snaps scrub ok 2024-02-10T05:53:29.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:29 smithi064 ceph-mon[38958]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-10T05:53:29.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:29 smithi064 ceph-mon[38958]: osd.4 [v2:172.21.15.176:6800/3371979738,v1:172.21.15.176:6801/3371979738] boot 2024-02-10T05:53:29.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:29 smithi064 ceph-mon[38958]: osdmap e30: 5 total, 5 up, 5 in 2024-02-10T05:53:29.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:29 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:53:30.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:30 smithi176 ceph-mon[30294]: osdmap e31: 5 total, 5 up, 5 in 2024-02-10T05:53:30.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:30 smithi064 ceph-mon[25185]: osdmap e31: 5 total, 5 up, 5 in 2024-02-10T05:53:30.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:30 smithi064 ceph-mon[38958]: osdmap e31: 5 total, 5 up, 5 in 2024-02-10T05:53:31.197 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:31 smithi176 ceph-mon[30294]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:53:31.198 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:31 smithi176 ceph-mon[30294]: osdmap e32: 5 total, 5 up, 5 in 2024-02-10T05:53:31.198 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:31 smithi176 ceph-mon[30294]: osdmap e33: 5 total, 5 up, 5 in 2024-02-10T05:53:31.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:31 smithi064 ceph-mon[25185]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:53:31.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:31 smithi064 ceph-mon[25185]: osdmap e32: 5 total, 5 up, 5 in 2024-02-10T05:53:31.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:31 smithi064 ceph-mon[25185]: osdmap e33: 5 total, 5 up, 5 in 2024-02-10T05:53:31.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:31 smithi064 ceph-mon[38958]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:53:31.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:31 smithi064 ceph-mon[38958]: osdmap e32: 5 total, 5 up, 5 in 2024-02-10T05:53:31.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:31 smithi064 ceph-mon[38958]: osdmap e33: 5 total, 5 up, 5 in 2024-02-10T05:53:32.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:32 smithi176 ceph-mon[30294]: Detected new or changed devices on smithi176 2024-02-10T05:53:32.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:32 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:32.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:32 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:32.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:32 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:32.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:32 smithi176 ceph-mon[30294]: Adjusting osd_memory_target on smithi176 to 17088M 2024-02-10T05:53:32.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:32 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:32.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:32 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:32.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:32 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:32.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:32 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:32.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:32 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:32.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:32 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:32.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:32 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:32.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:32 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:32.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:32 smithi176 ceph-mon[30294]: osdmap e34: 5 total, 5 up, 5 in 2024-02-10T05:53:32.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:32 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:53:32.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:32 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:53:32.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:32 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:32.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[25185]: Detected new or changed devices on smithi176 2024-02-10T05:53:32.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:32.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:32.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:32.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[25185]: Adjusting osd_memory_target on smithi176 to 17088M 2024-02-10T05:53:32.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:32.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:32.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:32.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:32.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:32.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:32.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:32.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:32.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[25185]: osdmap e34: 5 total, 5 up, 5 in 2024-02-10T05:53:32.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:53:32.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:53:32.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:32.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[38958]: Detected new or changed devices on smithi176 2024-02-10T05:53:32.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:32.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:32.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:32.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[38958]: Adjusting osd_memory_target on smithi176 to 17088M 2024-02-10T05:53:32.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:32.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:32.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:32.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:32.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:32.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:32.629 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:32.629 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:32.629 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[38958]: osdmap e34: 5 total, 5 up, 5 in 2024-02-10T05:53:32.629 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:53:32.629 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:53:32.629 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:32 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:33.641 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:33 smithi176 ceph-mon[30294]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:33.641 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:33 smithi176 ceph-mon[30294]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:53:33.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:33 smithi064 ceph-mon[25185]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:33.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:33 smithi064 ceph-mon[25185]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:53:33.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:33 smithi064 ceph-mon[38958]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:33.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:33 smithi064 ceph-mon[38958]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:53:34.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:34 smithi064 ceph-mon[25185]: from='client.? 172.21.15.176:0/242622404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd30b1ea-5f0f-4f4a-8f3c-c1c35514acd7"}]: dispatch 2024-02-10T05:53:34.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:34 smithi064 ceph-mon[25185]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd30b1ea-5f0f-4f4a-8f3c-c1c35514acd7"}]: dispatch 2024-02-10T05:53:34.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:34 smithi064 ceph-mon[25185]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd30b1ea-5f0f-4f4a-8f3c-c1c35514acd7"}]': finished 2024-02-10T05:53:34.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:34 smithi064 ceph-mon[25185]: osdmap e35: 6 total, 5 up, 6 in 2024-02-10T05:53:34.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:34 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:53:34.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:34 smithi064 ceph-mon[38958]: from='client.? 172.21.15.176:0/242622404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd30b1ea-5f0f-4f4a-8f3c-c1c35514acd7"}]: dispatch 2024-02-10T05:53:34.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:34 smithi064 ceph-mon[38958]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd30b1ea-5f0f-4f4a-8f3c-c1c35514acd7"}]: dispatch 2024-02-10T05:53:34.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:34 smithi064 ceph-mon[38958]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd30b1ea-5f0f-4f4a-8f3c-c1c35514acd7"}]': finished 2024-02-10T05:53:34.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:34 smithi064 ceph-mon[38958]: osdmap e35: 6 total, 5 up, 6 in 2024-02-10T05:53:34.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:34 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:53:34.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:34 smithi176 ceph-mon[30294]: from='client.? 172.21.15.176:0/242622404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd30b1ea-5f0f-4f4a-8f3c-c1c35514acd7"}]: dispatch 2024-02-10T05:53:34.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:34 smithi176 ceph-mon[30294]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dd30b1ea-5f0f-4f4a-8f3c-c1c35514acd7"}]: dispatch 2024-02-10T05:53:34.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:34 smithi176 ceph-mon[30294]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dd30b1ea-5f0f-4f4a-8f3c-c1c35514acd7"}]': finished 2024-02-10T05:53:34.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:34 smithi176 ceph-mon[30294]: osdmap e35: 6 total, 5 up, 6 in 2024-02-10T05:53:34.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:34 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:53:35.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:35 smithi064 ceph-mon[25185]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-02-10T05:53:35.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:35 smithi064 ceph-mon[38958]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-02-10T05:53:35.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:35 smithi176 ceph-mon[30294]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-02-10T05:53:36.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:36 smithi064 ceph-mon[25185]: from='client.? 172.21.15.176:0/4211892204' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:53:36.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:36 smithi064 ceph-mon[38958]: from='client.? 172.21.15.176:0/4211892204' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:53:36.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:36 smithi176 ceph-mon[30294]: from='client.? 172.21.15.176:0/4211892204' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:53:37.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:37 smithi064 ceph-mon[25185]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-02-10T05:53:37.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:37 smithi064 ceph-mon[38958]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-02-10T05:53:37.968 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:37 smithi176 ceph-mon[30294]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-02-10T05:53:39.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:39 smithi064 ceph-mon[25185]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-02-10T05:53:39.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:39 smithi064 ceph-mon[38958]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-02-10T05:53:39.956 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:39 smithi176 ceph-mon[30294]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-02-10T05:53:41.633 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:41 smithi176 ceph-mon[30294]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-02-10T05:53:41.633 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:41 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T05:53:41.633 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:41 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:41.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:41 smithi064 ceph-mon[25185]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-02-10T05:53:41.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:41 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T05:53:41.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:41 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:41.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:41 smithi064 ceph-mon[38958]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-02-10T05:53:41.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:41 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-10T05:53:41.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:41 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:42.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:42 smithi064 ceph-mon[25185]: Deploying daemon osd.5 on smithi176 2024-02-10T05:53:42.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:42 smithi064 ceph-mon[38958]: Deploying daemon osd.5 on smithi176 2024-02-10T05:53:42.944 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:42 smithi176 ceph-mon[30294]: Deploying daemon osd.5 on smithi176 2024-02-10T05:53:43.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:43 smithi064 ceph-mon[25185]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T05:53:43.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:43 smithi064 ceph-mon[38958]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T05:53:43.942 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:43 smithi176 ceph-mon[30294]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T05:53:45.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:45 smithi064 ceph-mon[25185]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:53:45.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:45 smithi064 ceph-mon[38958]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:53:45.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:45 smithi176 ceph-mon[30294]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:53:48.126 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:47 smithi176 ceph-mon[30294]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:53:48.126 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:47 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:48.126 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:47 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:48.126 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:47 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:48.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:47 smithi064 ceph-mon[25185]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:53:48.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:48.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:48.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:48.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:47 smithi064 ceph-mon[38958]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:53:48.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:47 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:48.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:47 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:48.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:47 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:50.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:49 smithi064 ceph-mon[25185]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:53:50.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:49 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:50.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:49 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:50.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:49 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:50.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:49 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:50.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:49 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:50.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:49 smithi064 ceph-mon[38958]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:53:50.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:49 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:50.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:49 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:50.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:49 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:50.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:49 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:50.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:49 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:50.152 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:49 smithi176 ceph-mon[30294]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:53:50.152 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:49 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:50.152 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:49 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:50.152 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:49 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:50.152 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:49 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:50.152 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:49 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:50.201 INFO:teuthology.orchestra.run.smithi176.stdout:Created osd(s) 5 on host 'smithi176' 2024-02-10T05:53:50.788 DEBUG:teuthology.orchestra.run.smithi176:osd.5> sudo journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.5.service 2024-02-10T05:53:50.790 INFO:tasks.cephadm:Deploying osd.6 on smithi176 with /dev/nvme2n1... 2024-02-10T05:53:50.791 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/nvme2n1 2024-02-10T05:53:51.095 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:50 smithi176 ceph-mon[30294]: from='osd.5 [v2:172.21.15.176:6808/2732967935,v1:172.21.15.176:6809/2732967935]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:53:51.096 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:50 smithi176 ceph-mon[30294]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:53:51.096 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:50 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:51.096 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:50 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:51.096 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:50 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:51.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:50 smithi064 ceph-mon[25185]: from='osd.5 [v2:172.21.15.176:6808/2732967935,v1:172.21.15.176:6809/2732967935]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:53:51.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:50 smithi064 ceph-mon[25185]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:53:51.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:50 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:51.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:50 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:51.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:50 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:51.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:50 smithi064 ceph-mon[38958]: from='osd.5 [v2:172.21.15.176:6808/2732967935,v1:172.21.15.176:6809/2732967935]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:53:51.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:50 smithi064 ceph-mon[38958]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-10T05:53:51.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:50 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:51.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:50 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:51.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:50 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:53:51.582 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:53:52.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:51 smithi064 ceph-mon[25185]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:53:52.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:51 smithi064 ceph-mon[25185]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T05:53:52.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:51 smithi064 ceph-mon[25185]: from='osd.5 [v2:172.21.15.176:6808/2732967935,v1:172.21.15.176:6809/2732967935]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-10T05:53:52.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:51 smithi064 ceph-mon[25185]: osdmap e36: 6 total, 5 up, 6 in 2024-02-10T05:53:52.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:51 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:53:52.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:51 smithi064 ceph-mon[25185]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-10T05:53:52.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:51 smithi064 ceph-mon[38958]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:53:52.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:51 smithi064 ceph-mon[38958]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T05:53:52.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:51 smithi064 ceph-mon[38958]: from='osd.5 [v2:172.21.15.176:6808/2732967935,v1:172.21.15.176:6809/2732967935]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-10T05:53:52.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:51 smithi064 ceph-mon[38958]: osdmap e36: 6 total, 5 up, 6 in 2024-02-10T05:53:52.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:51 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:53:52.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:51 smithi064 ceph-mon[38958]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-10T05:53:52.130 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:51 smithi176 ceph-mon[30294]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:53:52.131 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:51 smithi176 ceph-mon[30294]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-10T05:53:52.131 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:51 smithi176 ceph-mon[30294]: from='osd.5 [v2:172.21.15.176:6808/2732967935,v1:172.21.15.176:6809/2732967935]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-10T05:53:52.131 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:51 smithi176 ceph-mon[30294]: osdmap e36: 6 total, 5 up, 6 in 2024-02-10T05:53:52.131 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:51 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:53:52.131 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:51 smithi176 ceph-mon[30294]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-10T05:53:52.131 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:53:51 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:53:51.850+0000 7f5e0890d700 -1 osd.5 0 waiting for initial osdmap 2024-02-10T05:53:52.131 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:53:51 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:53:51.860+0000 7f5e04700700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:53:53.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:52 smithi064 ceph-mon[25185]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-02-10T05:53:53.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:52 smithi064 ceph-mon[25185]: osdmap e37: 6 total, 5 up, 6 in 2024-02-10T05:53:53.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:52 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:53:53.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:52 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:53:53.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:52 smithi064 ceph-mon[38958]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-02-10T05:53:53.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:52 smithi064 ceph-mon[38958]: osdmap e37: 6 total, 5 up, 6 in 2024-02-10T05:53:53.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:52 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:53:53.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:52 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:53:53.162 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:52 smithi176 ceph-mon[30294]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-02-10T05:53:53.162 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:52 smithi176 ceph-mon[30294]: osdmap e37: 6 total, 5 up, 6 in 2024-02-10T05:53:53.163 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:52 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:53:53.163 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:52 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:53:54.032 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping: /dev/nvme2n1 2024-02-10T05:53:54.032 INFO:teuthology.orchestra.run.smithi176.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-02-10T05:53:54.032 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme2n1 bs=1M count=10 conv=fsync 2024-02-10T05:53:54.032 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10+0 records in 2024-02-10T05:53:54.032 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2024-02-10T05:53:54.032 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.015939 s, 658 MB/s 2024-02-10T05:53:54.032 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping successful for: 2024-02-10T05:53:54.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:53 smithi064 ceph-mon[25185]: purged_snaps scrub starts 2024-02-10T05:53:54.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:53 smithi064 ceph-mon[25185]: purged_snaps scrub ok 2024-02-10T05:53:54.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:53 smithi064 ceph-mon[25185]: pgmap v131: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:53:54.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:53 smithi064 ceph-mon[25185]: osd.5 [v2:172.21.15.176:6808/2732967935,v1:172.21.15.176:6809/2732967935] boot 2024-02-10T05:53:54.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:53 smithi064 ceph-mon[25185]: osdmap e38: 6 total, 6 up, 6 in 2024-02-10T05:53:54.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:53 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:53:54.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:53 smithi064 ceph-mon[25185]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T05:53:54.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:53 smithi064 ceph-mon[25185]: osdmap e39: 6 total, 6 up, 6 in 2024-02-10T05:53:54.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:53 smithi064 ceph-mon[38958]: purged_snaps scrub starts 2024-02-10T05:53:54.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:53 smithi064 ceph-mon[38958]: purged_snaps scrub ok 2024-02-10T05:53:54.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:53 smithi064 ceph-mon[38958]: pgmap v131: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:53:54.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:53 smithi064 ceph-mon[38958]: osd.5 [v2:172.21.15.176:6808/2732967935,v1:172.21.15.176:6809/2732967935] boot 2024-02-10T05:53:54.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:53 smithi064 ceph-mon[38958]: osdmap e38: 6 total, 6 up, 6 in 2024-02-10T05:53:54.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:53 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:53:54.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:53 smithi064 ceph-mon[38958]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T05:53:54.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:53 smithi064 ceph-mon[38958]: osdmap e39: 6 total, 6 up, 6 in 2024-02-10T05:53:54.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:53 smithi176 ceph-mon[30294]: purged_snaps scrub starts 2024-02-10T05:53:54.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:53 smithi176 ceph-mon[30294]: purged_snaps scrub ok 2024-02-10T05:53:54.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:53 smithi176 ceph-mon[30294]: pgmap v131: 1 pgs: 1 peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-02-10T05:53:54.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:53 smithi176 ceph-mon[30294]: osd.5 [v2:172.21.15.176:6808/2732967935,v1:172.21.15.176:6809/2732967935] boot 2024-02-10T05:53:54.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:53 smithi176 ceph-mon[30294]: osdmap e38: 6 total, 6 up, 6 in 2024-02-10T05:53:54.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:53 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:53:54.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:53 smithi176 ceph-mon[30294]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-10T05:53:54.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:53 smithi176 ceph-mon[30294]: osdmap e39: 6 total, 6 up, 6 in 2024-02-10T05:53:54.671 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi176:/dev/nvme2n1 2024-02-10T05:53:55.253 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:53:55.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:55 smithi176 ceph-mon[30294]: pgmap v134: 1 pgs: 1 peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:53:55.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:55 smithi176 ceph-mon[30294]: osdmap e40: 6 total, 6 up, 6 in 2024-02-10T05:53:55.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:55 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:55.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:55 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:55.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:55 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:55.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:55 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:55.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:55 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:55.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:55 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:55.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:55 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:55.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:55 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:55.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:55 smithi176 ceph-mon[30294]: osdmap e41: 6 total, 6 up, 6 in 2024-02-10T05:53:56.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:55 smithi064 ceph-mon[25185]: pgmap v134: 1 pgs: 1 peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:53:56.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:55 smithi064 ceph-mon[25185]: osdmap e40: 6 total, 6 up, 6 in 2024-02-10T05:53:56.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:55 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:56.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:55 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:56.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:55 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:56.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:55 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:56.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:55 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:56.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:55 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:56.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:55 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:56.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:55 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:56.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:55 smithi064 ceph-mon[25185]: osdmap e41: 6 total, 6 up, 6 in 2024-02-10T05:53:56.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:55 smithi064 ceph-mon[38958]: pgmap v134: 1 pgs: 1 peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:53:56.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:55 smithi064 ceph-mon[38958]: osdmap e40: 6 total, 6 up, 6 in 2024-02-10T05:53:56.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:55 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:56.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:55 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:56.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:55 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:56.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:55 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:53:56.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:55 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:56.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:55 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:56.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:55 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:53:56.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:55 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:53:56.128 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:55 smithi064 ceph-mon[38958]: osdmap e41: 6 total, 6 up, 6 in 2024-02-10T05:53:56.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:56 smithi176 ceph-mon[30294]: Detected new or changed devices on smithi176 2024-02-10T05:53:56.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:56 smithi176 ceph-mon[30294]: Adjusting osd_memory_target on smithi176 to 8544M 2024-02-10T05:53:56.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:56 smithi176 ceph-mon[30294]: osdmap e42: 6 total, 6 up, 6 in 2024-02-10T05:53:57.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:56 smithi064 ceph-mon[25185]: Detected new or changed devices on smithi176 2024-02-10T05:53:57.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:56 smithi064 ceph-mon[25185]: Adjusting osd_memory_target on smithi176 to 8544M 2024-02-10T05:53:57.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:56 smithi064 ceph-mon[25185]: osdmap e42: 6 total, 6 up, 6 in 2024-02-10T05:53:57.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:56 smithi064 ceph-mon[38958]: Detected new or changed devices on smithi176 2024-02-10T05:53:57.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:56 smithi064 ceph-mon[38958]: Adjusting osd_memory_target on smithi176 to 8544M 2024-02-10T05:53:57.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:56 smithi064 ceph-mon[38958]: osdmap e42: 6 total, 6 up, 6 in 2024-02-10T05:53:58.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:57 smithi064 ceph-mon[25185]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:53:58.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:57 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:53:58.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:57 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:53:58.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:57 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:58.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:57 smithi064 ceph-mon[38958]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:53:58.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:57 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:53:58.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:57 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:53:58.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:57 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:58.163 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:57 smithi176 ceph-mon[30294]: pgmap v137: 1 pgs: 1 peering; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:53:58.163 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:57 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:53:58.163 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:57 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:53:58.163 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:57 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:53:58.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:58 smithi176 ceph-mon[30294]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:59.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:58 smithi064 ceph-mon[25185]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:53:59.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:58 smithi064 ceph-mon[38958]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:00.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:59 smithi176 ceph-mon[30294]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-02-10T05:54:00.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:59 smithi176 ceph-mon[30294]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T05:54:00.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:59 smithi176 ceph-mon[30294]: Cluster is now healthy 2024-02-10T05:54:00.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:59 smithi176 ceph-mon[30294]: from='client.? 172.21.15.176:0/1263235148' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "093e0c53-f0e6-43c3-9d9d-141bc456784d"}]: dispatch 2024-02-10T05:54:00.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:59 smithi176 ceph-mon[30294]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "093e0c53-f0e6-43c3-9d9d-141bc456784d"}]: dispatch 2024-02-10T05:54:00.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:59 smithi176 ceph-mon[30294]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "093e0c53-f0e6-43c3-9d9d-141bc456784d"}]': finished 2024-02-10T05:54:00.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:59 smithi176 ceph-mon[30294]: osdmap e43: 7 total, 6 up, 7 in 2024-02-10T05:54:00.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:53:59 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:54:00.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:59 smithi064 ceph-mon[25185]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-02-10T05:54:00.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:59 smithi064 ceph-mon[25185]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T05:54:00.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:59 smithi064 ceph-mon[25185]: Cluster is now healthy 2024-02-10T05:54:00.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:59 smithi064 ceph-mon[25185]: from='client.? 172.21.15.176:0/1263235148' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "093e0c53-f0e6-43c3-9d9d-141bc456784d"}]: dispatch 2024-02-10T05:54:00.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:59 smithi064 ceph-mon[25185]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "093e0c53-f0e6-43c3-9d9d-141bc456784d"}]: dispatch 2024-02-10T05:54:00.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:59 smithi064 ceph-mon[25185]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "093e0c53-f0e6-43c3-9d9d-141bc456784d"}]': finished 2024-02-10T05:54:00.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:59 smithi064 ceph-mon[25185]: osdmap e43: 7 total, 6 up, 7 in 2024-02-10T05:54:00.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:53:59 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:54:00.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:59 smithi064 ceph-mon[38958]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail; 141 KiB/s, 0 objects/s recovering 2024-02-10T05:54:00.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:59 smithi064 ceph-mon[38958]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-10T05:54:00.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:59 smithi064 ceph-mon[38958]: Cluster is now healthy 2024-02-10T05:54:00.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:59 smithi064 ceph-mon[38958]: from='client.? 172.21.15.176:0/1263235148' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "093e0c53-f0e6-43c3-9d9d-141bc456784d"}]: dispatch 2024-02-10T05:54:00.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:59 smithi064 ceph-mon[38958]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "093e0c53-f0e6-43c3-9d9d-141bc456784d"}]: dispatch 2024-02-10T05:54:00.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:59 smithi064 ceph-mon[38958]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "093e0c53-f0e6-43c3-9d9d-141bc456784d"}]': finished 2024-02-10T05:54:00.378 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:59 smithi064 ceph-mon[38958]: osdmap e43: 7 total, 6 up, 7 in 2024-02-10T05:54:00.378 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:53:59 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:54:02.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:01 smithi176 ceph-mon[30294]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-02-10T05:54:02.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:01 smithi176 ceph-mon[30294]: from='client.? 172.21.15.176:0/808777264' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:54:02.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:01 smithi064 ceph-mon[25185]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-02-10T05:54:02.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:01 smithi064 ceph-mon[25185]: from='client.? 172.21.15.176:0/808777264' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:54:02.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:01 smithi064 ceph-mon[38958]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-02-10T05:54:02.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:01 smithi064 ceph-mon[38958]: from='client.? 172.21.15.176:0/808777264' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:54:03.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:03 smithi176 ceph-mon[30294]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-02-10T05:54:04.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:03 smithi064 ceph-mon[25185]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-02-10T05:54:04.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:03 smithi064 ceph-mon[38958]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-02-10T05:54:06.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:05 smithi176 ceph-mon[30294]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-10T05:54:06.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:05 smithi064 ceph-mon[25185]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-10T05:54:06.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:05 smithi064 ceph-mon[38958]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-10T05:54:07.160 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:06 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T05:54:07.160 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:06 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:07.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:06 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T05:54:07.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:06 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:07.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-10T05:54:07.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:06 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:08.062 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:07 smithi176 ceph-mon[30294]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-02-10T05:54:08.062 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:07 smithi176 ceph-mon[30294]: Deploying daemon osd.6 on smithi176 2024-02-10T05:54:08.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:07 smithi064 ceph-mon[25185]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-02-10T05:54:08.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:07 smithi064 ceph-mon[25185]: Deploying daemon osd.6 on smithi176 2024-02-10T05:54:08.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:07 smithi064 ceph-mon[38958]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-02-10T05:54:08.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:07 smithi064 ceph-mon[38958]: Deploying daemon osd.6 on smithi176 2024-02-10T05:54:09.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:08 smithi176 ceph-mon[30294]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:54:09.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:08 smithi064 ceph-mon[25185]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:54:09.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:08 smithi064 ceph-mon[38958]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:54:11.801 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:11 smithi176 ceph-mon[30294]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:54:11.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:11 smithi064 ceph-mon[25185]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:54:11.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:11 smithi064 ceph-mon[38958]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:54:13.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:13 smithi064 ceph-mon[25185]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:54:13.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:13 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:13.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:13 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:13.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:13 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:13.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:13 smithi064 ceph-mon[38958]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:54:13.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:13 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:13.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:13 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:13.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:13 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:13.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:13 smithi176 ceph-mon[30294]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:54:13.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:13 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:13.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:13 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:13.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:13 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:15.328 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:15 smithi176 ceph-mon[30294]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:54:15.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:15 smithi064 ceph-mon[25185]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:54:15.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:15 smithi064 ceph-mon[38958]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:54:15.703 INFO:teuthology.orchestra.run.smithi176.stdout:Created osd(s) 6 on host 'smithi176' 2024-02-10T05:54:16.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:16 smithi064 ceph-mon[25185]: from='osd.6 [v2:172.21.15.176:6816/937306186,v1:172.21.15.176:6817/937306186]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:54:16.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:16 smithi064 ceph-mon[25185]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:54:16.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:16 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:16.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:16 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:16.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:16 smithi064 ceph-mon[38958]: from='osd.6 [v2:172.21.15.176:6816/937306186,v1:172.21.15.176:6817/937306186]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:54:16.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:16 smithi064 ceph-mon[38958]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:54:16.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:16 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:16.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:16 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:16.423 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:16 smithi176 ceph-mon[30294]: from='osd.6 [v2:172.21.15.176:6816/937306186,v1:172.21.15.176:6817/937306186]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:54:16.424 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:16 smithi176 ceph-mon[30294]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-10T05:54:16.424 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:16 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:16.424 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:16 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:16.452 DEBUG:teuthology.orchestra.run.smithi176:osd.6> sudo journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.6.service 2024-02-10T05:54:16.455 INFO:tasks.cephadm:Deploying osd.7 on smithi176 with /dev/nvme1n1... 2024-02-10T05:54:16.455 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/nvme1n1 2024-02-10T05:54:16.747 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:54:17.152 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:17 smithi176 ceph-mon[30294]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:54:17.152 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:17 smithi176 ceph-mon[30294]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T05:54:17.152 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:17 smithi176 ceph-mon[30294]: osdmap e44: 7 total, 6 up, 7 in 2024-02-10T05:54:17.152 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:17 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:54:17.152 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:17 smithi176 ceph-mon[30294]: from='osd.6 [v2:172.21.15.176:6816/937306186,v1:172.21.15.176:6817/937306186]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-10T05:54:17.153 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:17 smithi176 ceph-mon[30294]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-10T05:54:17.153 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:17 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:17.153 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:17 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:17.153 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:54:17 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:54:17.081+0000 7f1c993be700 -1 osd.6 0 waiting for initial osdmap 2024-02-10T05:54:17.153 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:54:17 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:54:17.087+0000 7f1c951b1700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:54:17.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:17 smithi064 ceph-mon[25185]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:54:17.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:17 smithi064 ceph-mon[25185]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T05:54:17.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:17 smithi064 ceph-mon[25185]: osdmap e44: 7 total, 6 up, 7 in 2024-02-10T05:54:17.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:17 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:54:17.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:17 smithi064 ceph-mon[25185]: from='osd.6 [v2:172.21.15.176:6816/937306186,v1:172.21.15.176:6817/937306186]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-10T05:54:17.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:17 smithi064 ceph-mon[25185]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-10T05:54:17.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:17 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:17.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:17 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:17.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:17 smithi064 ceph-mon[38958]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:54:17.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:17 smithi064 ceph-mon[38958]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-10T05:54:17.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:17 smithi064 ceph-mon[38958]: osdmap e44: 7 total, 6 up, 7 in 2024-02-10T05:54:17.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:17 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:54:17.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:17 smithi064 ceph-mon[38958]: from='osd.6 [v2:172.21.15.176:6816/937306186,v1:172.21.15.176:6817/937306186]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-10T05:54:17.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:17 smithi064 ceph-mon[38958]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-10T05:54:17.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:17 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:17.378 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:17 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:18.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:18 smithi064 ceph-mon[25185]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-02-10T05:54:18.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:18 smithi064 ceph-mon[25185]: osdmap e45: 7 total, 6 up, 7 in 2024-02-10T05:54:18.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:18 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:54:18.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:18 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:54:18.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:18 smithi064 ceph-mon[38958]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-02-10T05:54:18.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:18 smithi064 ceph-mon[38958]: osdmap e45: 7 total, 6 up, 7 in 2024-02-10T05:54:18.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:18 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:54:18.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:18 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:54:18.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:18 smithi176 ceph-mon[30294]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-02-10T05:54:18.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:18 smithi176 ceph-mon[30294]: osdmap e45: 7 total, 6 up, 7 in 2024-02-10T05:54:18.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:18 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:54:18.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:18 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:54:19.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:19 smithi176 ceph-mon[30294]: purged_snaps scrub starts 2024-02-10T05:54:19.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:19 smithi176 ceph-mon[30294]: purged_snaps scrub ok 2024-02-10T05:54:19.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:19 smithi176 ceph-mon[30294]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:54:19.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:19 smithi176 ceph-mon[30294]: osd.6 [v2:172.21.15.176:6816/937306186,v1:172.21.15.176:6817/937306186] boot 2024-02-10T05:54:19.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:19 smithi176 ceph-mon[30294]: osdmap e46: 7 total, 7 up, 7 in 2024-02-10T05:54:19.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:19 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:54:19.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:19 smithi176 ceph-mon[30294]: osdmap e47: 7 total, 7 up, 7 in 2024-02-10T05:54:19.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:19 smithi064 ceph-mon[38958]: purged_snaps scrub starts 2024-02-10T05:54:19.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:19 smithi064 ceph-mon[38958]: purged_snaps scrub ok 2024-02-10T05:54:19.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:19 smithi064 ceph-mon[38958]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:54:19.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:19 smithi064 ceph-mon[38958]: osd.6 [v2:172.21.15.176:6816/937306186,v1:172.21.15.176:6817/937306186] boot 2024-02-10T05:54:19.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:19 smithi064 ceph-mon[38958]: osdmap e46: 7 total, 7 up, 7 in 2024-02-10T05:54:19.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:19 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:54:19.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:19 smithi064 ceph-mon[38958]: osdmap e47: 7 total, 7 up, 7 in 2024-02-10T05:54:19.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:19 smithi064 ceph-mon[25185]: purged_snaps scrub starts 2024-02-10T05:54:19.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:19 smithi064 ceph-mon[25185]: purged_snaps scrub ok 2024-02-10T05:54:19.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:19 smithi064 ceph-mon[25185]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-02-10T05:54:19.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:19 smithi064 ceph-mon[25185]: osd.6 [v2:172.21.15.176:6816/937306186,v1:172.21.15.176:6817/937306186] boot 2024-02-10T05:54:19.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:19 smithi064 ceph-mon[25185]: osdmap e46: 7 total, 7 up, 7 in 2024-02-10T05:54:19.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:19 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:54:19.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:19 smithi064 ceph-mon[25185]: osdmap e47: 7 total, 7 up, 7 in 2024-02-10T05:54:19.730 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping: /dev/nvme1n1 2024-02-10T05:54:19.730 INFO:teuthology.orchestra.run.smithi176.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-02-10T05:54:19.731 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme1n1 bs=1M count=10 conv=fsync 2024-02-10T05:54:19.731 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10+0 records in 2024-02-10T05:54:19.731 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2024-02-10T05:54:19.731 INFO:teuthology.orchestra.run.smithi176.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0132413 s, 792 MB/s 2024-02-10T05:54:19.731 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping successful for: 2024-02-10T05:54:20.408 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi176:/dev/nvme1n1 2024-02-10T05:54:20.659 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:54:21.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:21 smithi064 ceph-mon[25185]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:54:21.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:21 smithi064 ceph-mon[25185]: osdmap e48: 7 total, 7 up, 7 in 2024-02-10T05:54:21.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:21 smithi064 ceph-mon[25185]: osdmap e49: 7 total, 7 up, 7 in 2024-02-10T05:54:21.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:21 smithi064 ceph-mon[38958]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:54:21.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:21 smithi064 ceph-mon[38958]: osdmap e48: 7 total, 7 up, 7 in 2024-02-10T05:54:21.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:21 smithi064 ceph-mon[38958]: osdmap e49: 7 total, 7 up, 7 in 2024-02-10T05:54:21.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:21 smithi176 ceph-mon[30294]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:54:21.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:21 smithi176 ceph-mon[30294]: osdmap e48: 7 total, 7 up, 7 in 2024-02-10T05:54:21.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:21 smithi176 ceph-mon[30294]: osdmap e49: 7 total, 7 up, 7 in 2024-02-10T05:54:22.663 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:22 smithi176 ceph-mon[30294]: Detected new or changed devices on smithi176 2024-02-10T05:54:22.663 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:22 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:22.663 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:22 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:22.663 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:22 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:22.663 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:22 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:22.663 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:22 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:22.663 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:22 smithi176 ceph-mon[30294]: Adjusting osd_memory_target on smithi176 to 5696M 2024-02-10T05:54:22.663 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:22 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:22.663 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:22 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:22.663 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:22 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:22.664 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:22 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:22.664 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:22 smithi176 ceph-mon[30294]: osdmap e50: 7 total, 7 up, 7 in 2024-02-10T05:54:22.664 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:22 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:22.664 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:22 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:22.664 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:22 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:22.664 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:22 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:22.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[25185]: Detected new or changed devices on smithi176 2024-02-10T05:54:22.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:22.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:22.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:22.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:22.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:22.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[25185]: Adjusting osd_memory_target on smithi176 to 5696M 2024-02-10T05:54:22.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:22.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:22.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:22.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:22.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[25185]: osdmap e50: 7 total, 7 up, 7 in 2024-02-10T05:54:22.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:22.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:22.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:22.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:22.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[38958]: Detected new or changed devices on smithi176 2024-02-10T05:54:22.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:22.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:22.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:22.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:22.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:22.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[38958]: Adjusting osd_memory_target on smithi176 to 5696M 2024-02-10T05:54:22.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:22.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:22.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:22.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:22.879 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[38958]: osdmap e50: 7 total, 7 up, 7 in 2024-02-10T05:54:22.879 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:22.879 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:22.879 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:22.879 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:22 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:23.971 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:23 smithi176 ceph-mon[30294]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:54:23.971 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:23 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:54:23.972 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:23 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:54:23.972 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:23 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:24.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:23 smithi064 ceph-mon[25185]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:54:24.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:23 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:54:24.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:23 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:54:24.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:23 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:24.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:23 smithi064 ceph-mon[38958]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:54:24.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:23 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:54:24.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:23 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:54:24.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:23 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:24.872 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:24 smithi176 ceph-mon[30294]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:25.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:24 smithi064 ceph-mon[25185]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:25.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:24 smithi064 ceph-mon[38958]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:25.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:25 smithi176 ceph-mon[30294]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-02-10T05:54:26.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:25 smithi064 ceph-mon[38958]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-02-10T05:54:26.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:25 smithi064 ceph-mon[25185]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-02-10T05:54:26.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:26 smithi176 ceph-mon[30294]: from='client.? 172.21.15.176:0/1973869308' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef5d6516-1760-43db-bc37-a262e81466a1"}]: dispatch 2024-02-10T05:54:26.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:26 smithi176 ceph-mon[30294]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef5d6516-1760-43db-bc37-a262e81466a1"}]: dispatch 2024-02-10T05:54:26.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:26 smithi176 ceph-mon[30294]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef5d6516-1760-43db-bc37-a262e81466a1"}]': finished 2024-02-10T05:54:26.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:26 smithi176 ceph-mon[30294]: osdmap e51: 8 total, 7 up, 8 in 2024-02-10T05:54:26.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:26 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:54:27.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:26 smithi064 ceph-mon[25185]: from='client.? 172.21.15.176:0/1973869308' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef5d6516-1760-43db-bc37-a262e81466a1"}]: dispatch 2024-02-10T05:54:27.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:26 smithi064 ceph-mon[25185]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef5d6516-1760-43db-bc37-a262e81466a1"}]: dispatch 2024-02-10T05:54:27.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:26 smithi064 ceph-mon[25185]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef5d6516-1760-43db-bc37-a262e81466a1"}]': finished 2024-02-10T05:54:27.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:26 smithi064 ceph-mon[25185]: osdmap e51: 8 total, 7 up, 8 in 2024-02-10T05:54:27.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:26 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:54:27.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:26 smithi064 ceph-mon[38958]: from='client.? 172.21.15.176:0/1973869308' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef5d6516-1760-43db-bc37-a262e81466a1"}]: dispatch 2024-02-10T05:54:27.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:26 smithi064 ceph-mon[38958]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef5d6516-1760-43db-bc37-a262e81466a1"}]: dispatch 2024-02-10T05:54:27.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:26 smithi064 ceph-mon[38958]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef5d6516-1760-43db-bc37-a262e81466a1"}]': finished 2024-02-10T05:54:27.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:26 smithi064 ceph-mon[38958]: osdmap e51: 8 total, 7 up, 8 in 2024-02-10T05:54:27.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:26 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:54:28.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:27 smithi064 ceph-mon[25185]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-02-10T05:54:28.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:27 smithi064 ceph-mon[25185]: from='client.? 172.21.15.176:0/770959799' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:54:28.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:27 smithi064 ceph-mon[38958]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-02-10T05:54:28.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:27 smithi064 ceph-mon[38958]: from='client.? 172.21.15.176:0/770959799' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:54:28.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:27 smithi176 ceph-mon[30294]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-02-10T05:54:28.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:27 smithi176 ceph-mon[30294]: from='client.? 172.21.15.176:0/770959799' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:54:29.886 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:29 smithi176 ceph-mon[30294]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-02-10T05:54:30.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:29 smithi064 ceph-mon[25185]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-02-10T05:54:30.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:29 smithi064 ceph-mon[38958]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-02-10T05:54:31.962 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:31 smithi176 ceph-mon[30294]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-02-10T05:54:32.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:31 smithi064 ceph-mon[25185]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-02-10T05:54:32.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:31 smithi064 ceph-mon[38958]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-02-10T05:54:33.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:32 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T05:54:33.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:32 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:33.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:32 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T05:54:33.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:32 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:33.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:32 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-10T05:54:33.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:32 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:33.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:33 smithi176 ceph-mon[30294]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T05:54:33.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:33 smithi176 ceph-mon[30294]: Deploying daemon osd.7 on smithi176 2024-02-10T05:54:34.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:33 smithi064 ceph-mon[25185]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T05:54:34.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:33 smithi064 ceph-mon[25185]: Deploying daemon osd.7 on smithi176 2024-02-10T05:54:34.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:33 smithi064 ceph-mon[38958]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-10T05:54:34.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:33 smithi064 ceph-mon[38958]: Deploying daemon osd.7 on smithi176 2024-02-10T05:54:35.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:35 smithi176 ceph-mon[30294]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:54:36.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:35 smithi064 ceph-mon[25185]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:54:36.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:35 smithi064 ceph-mon[38958]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:54:37.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:37 smithi176 ceph-mon[30294]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:54:38.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:37 smithi064 ceph-mon[25185]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:54:38.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:37 smithi064 ceph-mon[38958]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:54:39.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:39 smithi176 ceph-mon[30294]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:54:39.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:39 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:39.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:39 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:39.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:39 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:39.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:39 smithi064 ceph-mon[25185]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:54:39.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:39 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:39.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:39 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:39.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:39 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:39.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:39 smithi064 ceph-mon[38958]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:54:39.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:39 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:39.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:39 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:39.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:39 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:41.650 INFO:teuthology.orchestra.run.smithi176.stdout:Created osd(s) 7 on host 'smithi176' 2024-02-10T05:54:41.853 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:41 smithi176 ceph-mon[30294]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:54:41.854 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:41 smithi176 ceph-mon[30294]: from='osd.7 [v2:172.21.15.176:6824/3087207983,v1:172.21.15.176:6825/3087207983]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:54:41.854 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:41 smithi176 ceph-mon[30294]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:54:41.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:41 smithi064 ceph-mon[25185]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:54:41.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:41 smithi064 ceph-mon[25185]: from='osd.7 [v2:172.21.15.176:6824/3087207983,v1:172.21.15.176:6825/3087207983]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:54:41.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:41 smithi064 ceph-mon[25185]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:54:41.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:41 smithi064 ceph-mon[38958]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:54:41.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:41 smithi064 ceph-mon[38958]: from='osd.7 [v2:172.21.15.176:6824/3087207983,v1:172.21.15.176:6825/3087207983]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:54:41.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:41 smithi064 ceph-mon[38958]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-10T05:54:42.245 DEBUG:teuthology.orchestra.run.smithi176:osd.7> sudo journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.7.service 2024-02-10T05:54:42.247 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-02-10T05:54:42.248 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph osd stat -f json 2024-02-10T05:54:42.505 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:54:42.866 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:42 smithi064 ceph-mon[25185]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T05:54:42.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:42 smithi064 ceph-mon[25185]: osdmap e52: 8 total, 7 up, 8 in 2024-02-10T05:54:42.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:42 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:54:42.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:42 smithi064 ceph-mon[25185]: from='osd.7 [v2:172.21.15.176:6824/3087207983,v1:172.21.15.176:6825/3087207983]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-10T05:54:42.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:42 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:42.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:42 smithi064 ceph-mon[25185]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-10T05:54:42.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:42 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:42.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:42 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:42.867 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:42 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:42.867 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:42 smithi064 ceph-mon[38958]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T05:54:42.868 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:42 smithi064 ceph-mon[38958]: osdmap e52: 8 total, 7 up, 8 in 2024-02-10T05:54:42.868 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:42 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:54:42.868 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:42 smithi064 ceph-mon[38958]: from='osd.7 [v2:172.21.15.176:6824/3087207983,v1:172.21.15.176:6825/3087207983]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-10T05:54:42.868 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:42 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:42.868 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:42 smithi064 ceph-mon[38958]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-10T05:54:42.868 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:42 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:42.868 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:42 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:42.868 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:42 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:42.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:42 smithi176 ceph-mon[30294]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-10T05:54:42.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:42 smithi176 ceph-mon[30294]: osdmap e52: 8 total, 7 up, 8 in 2024-02-10T05:54:42.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:42 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:54:42.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:42 smithi176 ceph-mon[30294]: from='osd.7 [v2:172.21.15.176:6824/3087207983,v1:172.21.15.176:6825/3087207983]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-10T05:54:42.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:42 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:42.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:42 smithi176 ceph-mon[30294]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-02-10T05:54:42.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:42 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:42.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:42 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:42.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:42 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:43.488 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:54:43 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:54:43.059+0000 7f8a4463d700 -1 osd.7 0 waiting for initial osdmap 2024-02-10T05:54:43.489 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:54:43 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:54:43.065+0000 7f8a3bc27700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:54:43.812 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:43 smithi176 ceph-mon[30294]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:54:43.813 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:43 smithi176 ceph-mon[30294]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-02-10T05:54:43.813 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:43 smithi176 ceph-mon[30294]: osdmap e53: 8 total, 7 up, 8 in 2024-02-10T05:54:43.813 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:43 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:54:43.813 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:43 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:54:43.813 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:43 smithi176 ceph-mon[30294]: from='osd.7 ' entity='osd.7' 2024-02-10T05:54:43.833 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:43 smithi064 ceph-mon[25185]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:54:43.834 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:43 smithi064 ceph-mon[25185]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-02-10T05:54:43.834 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:43 smithi064 ceph-mon[25185]: osdmap e53: 8 total, 7 up, 8 in 2024-02-10T05:54:43.834 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:43 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:54:43.834 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:43 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:54:43.834 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:43 smithi064 ceph-mon[25185]: from='osd.7 ' entity='osd.7' 2024-02-10T05:54:43.834 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:43 smithi064 ceph-mon[38958]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:54:43.834 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:43 smithi064 ceph-mon[38958]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-02-10T05:54:43.834 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:43 smithi064 ceph-mon[38958]: osdmap e53: 8 total, 7 up, 8 in 2024-02-10T05:54:43.834 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:43 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:54:43.834 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:43 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:54:43.834 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:43 smithi064 ceph-mon[38958]: from='osd.7 ' entity='osd.7' 2024-02-10T05:54:44.263 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:54:44.770 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:44 smithi176 ceph-mon[30294]: purged_snaps scrub starts 2024-02-10T05:54:44.770 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:44 smithi176 ceph-mon[30294]: purged_snaps scrub ok 2024-02-10T05:54:44.770 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:44 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:54:44.770 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:44 smithi176 ceph-mon[30294]: osd.7 [v2:172.21.15.176:6824/3087207983,v1:172.21.15.176:6825/3087207983] boot 2024-02-10T05:54:44.770 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:44 smithi176 ceph-mon[30294]: osdmap e54: 8 total, 8 up, 8 in 2024-02-10T05:54:44.770 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:44 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:54:44.770 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:44 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/2758174110' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T05:54:44.860 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:44 smithi064 ceph-mon[25185]: purged_snaps scrub starts 2024-02-10T05:54:44.861 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:44 smithi064 ceph-mon[25185]: purged_snaps scrub ok 2024-02-10T05:54:44.861 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:44 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:54:44.861 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:44 smithi064 ceph-mon[25185]: osd.7 [v2:172.21.15.176:6824/3087207983,v1:172.21.15.176:6825/3087207983] boot 2024-02-10T05:54:44.861 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:44 smithi064 ceph-mon[25185]: osdmap e54: 8 total, 8 up, 8 in 2024-02-10T05:54:44.861 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:44 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:54:44.861 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:44 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/2758174110' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T05:54:44.861 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:44 smithi064 ceph-mon[38958]: purged_snaps scrub starts 2024-02-10T05:54:44.861 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:44 smithi064 ceph-mon[38958]: purged_snaps scrub ok 2024-02-10T05:54:44.861 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:44 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:54:44.861 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:44 smithi064 ceph-mon[38958]: osd.7 [v2:172.21.15.176:6824/3087207983,v1:172.21.15.176:6825/3087207983] boot 2024-02-10T05:54:44.862 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:44 smithi064 ceph-mon[38958]: osdmap e54: 8 total, 8 up, 8 in 2024-02-10T05:54:44.862 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:44 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:54:44.862 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:44 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/2758174110' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T05:54:44.887 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1707544483,"num_in_osds":8,"osd_in_since":1707544465,"num_remapped_pgs":1} 2024-02-10T05:54:44.888 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T05:54:45.132 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:54:45.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:45 smithi064 ceph-mon[25185]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:54:45.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:45 smithi064 ceph-mon[25185]: osdmap e55: 8 total, 8 up, 8 in 2024-02-10T05:54:45.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:45 smithi064 ceph-mon[38958]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:54:45.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:45 smithi064 ceph-mon[38958]: osdmap e55: 8 total, 8 up, 8 in 2024-02-10T05:54:45.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:45 smithi176 ceph-mon[30294]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-02-10T05:54:45.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:45 smithi176 ceph-mon[30294]: osdmap e55: 8 total, 8 up, 8 in 2024-02-10T05:54:46.738 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:46 smithi176 ceph-mon[30294]: osdmap e56: 8 total, 8 up, 8 in 2024-02-10T05:54:46.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:46 smithi064 ceph-mon[25185]: osdmap e56: 8 total, 8 up, 8 in 2024-02-10T05:54:46.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:46 smithi064 ceph-mon[38958]: osdmap e56: 8 total, 8 up, 8 in 2024-02-10T05:54:46.883 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:54:46.884 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":57,"fsid":"00e86dbe-c7d8-11ee-95b8-87774f69a715","created":"2024-02-10T05:49:26.002096+0000","modified":"2024-02-10T05:54:46.629856+0000","last_up_change":"2024-02-10T05:54:43.610359+0000","last_in_change":"2024-02-10T05:54:25.763533+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T05:52:38.253691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"32df1361-a86f-44aa-bc33-d0b1e3fc140b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3211593074},{"type":"v1","addr":"172.21.15.64:6803","nonce":3211593074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3211593074},{"type":"v1","addr":"172.21.15.64:6805","nonce":3211593074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3211593074},{"type":"v1","addr":"172.21.15.64:6809","nonce":3211593074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3211593074},{"type":"v1","addr":"172.21.15.64:6807","nonce":3211593074}]},"public_addr":"172.21.15.64:6803/3211593074","cluster_addr":"172.21.15.64:6805/3211593074","heartbeat_back_addr":"172.21.15.64:6809/3211593074","heartbeat_front_addr":"172.21.15.64:6807/3211593074","state":["exists","up"]},{"osd":1,"uuid":"cfe981c9-406d-4da9-89ff-81dbbef91183","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":614169574},{"type":"v1","addr":"172.21.15.64:6811","nonce":614169574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":614169574},{"type":"v1","addr":"172.21.15.64:6813","nonce":614169574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":614169574},{"type":"v1","addr":"172.21.15.64:6817","nonce":614169574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":614169574},{"type":"v1","addr":"172.21.15.64:6815","nonce":614169574}]},"public_addr":"172.21.15.64:6811/614169574","cluster_addr":"172.21.15.64:6813/614169574","heartbeat_back_addr":"172.21.15.64:6817/614169574","heartbeat_front_addr":"172.21.15.64:6815/614169574","state":["exists","up"]},{"osd":2,"uuid":"bb433070-a584-427e-bb43-bf20be79b123","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2362209772},{"type":"v1","addr":"172.21.15.64:6819","nonce":2362209772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2362209772},{"type":"v1","addr":"172.21.15.64:6821","nonce":2362209772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":2362209772},{"type":"v1","addr":"172.21.15.64:6825","nonce":2362209772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2362209772},{"type":"v1","addr":"172.21.15.64:6823","nonce":2362209772}]},"public_addr":"172.21.15.64:6819/2362209772","cluster_addr":"172.21.15.64:6821/2362209772","heartbeat_back_addr":"172.21.15.64:6825/2362209772","heartbeat_front_addr":"172.21.15.64:6823/2362209772","state":["exists","up"]},{"osd":3,"uuid":"bb31284d-6c1a-4f79-928a-bb758b45e196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":256208898},{"type":"v1","addr":"172.21.15.64:6827","nonce":256208898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":256208898},{"type":"v1","addr":"172.21.15.64:6829","nonce":256208898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":256208898},{"type":"v1","addr":"172.21.15.64:6833","nonce":256208898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":256208898},{"type":"v1","addr":"172.21.15.64:6831","nonce":256208898}]},"public_addr":"172.21.15.64:6827/256208898","cluster_addr":"172.21.15.64:6829/256208898","heartbeat_back_addr":"172.21.15.64:6833/256208898","heartbeat_front_addr":"172.21.15.64:6831/256208898","state":["exists","up"]},{"osd":4,"uuid":"1a5c7c64-fbb4-4d8b-b3d5-144164369304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":3371979738},{"type":"v1","addr":"172.21.15.176:6801","nonce":3371979738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":3371979738},{"type":"v1","addr":"172.21.15.176:6803","nonce":3371979738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":3371979738},{"type":"v1","addr":"172.21.15.176:6807","nonce":3371979738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":3371979738},{"type":"v1","addr":"172.21.15.176:6805","nonce":3371979738}]},"public_addr":"172.21.15.176:6801/3371979738","cluster_addr":"172.21.15.176:6803/3371979738","heartbeat_back_addr":"172.21.15.176:6807/3371979738","heartbeat_front_addr":"172.21.15.176:6805/3371979738","state":["exists","up"]},{"osd":5,"uuid":"dd30b1ea-5f0f-4f4a-8f3c-c1c35514acd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":2732967935},{"type":"v1","addr":"172.21.15.176:6809","nonce":2732967935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2732967935},{"type":"v1","addr":"172.21.15.176:6811","nonce":2732967935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2732967935},{"type":"v1","addr":"172.21.15.176:6815","nonce":2732967935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2732967935},{"type":"v1","addr":"172.21.15.176:6813","nonce":2732967935}]},"public_addr":"172.21.15.176:6809/2732967935","cluster_addr":"172.21.15.176:6811/2732967935","heartbeat_back_addr":"172.21.15.176:6815/2732967935","heartbeat_front_addr":"172.21.15.176:6813/2732967935","state":["exists","up"]},{"osd":6,"uuid":"093e0c53-f0e6-43c3-9d9d-141bc456784d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":937306186},{"type":"v1","addr":"172.21.15.176:6817","nonce":937306186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":937306186},{"type":"v1","addr":"172.21.15.176:6819","nonce":937306186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":937306186},{"type":"v1","addr":"172.21.15.176:6823","nonce":937306186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":937306186},{"type":"v1","addr":"172.21.15.176:6821","nonce":937306186}]},"public_addr":"172.21.15.176:6817/937306186","cluster_addr":"172.21.15.176:6819/937306186","heartbeat_back_addr":"172.21.15.176:6823/937306186","heartbeat_front_addr":"172.21.15.176:6821/937306186","state":["exists","up"]},{"osd":7,"uuid":"ef5d6516-1760-43db-bc37-a262e81466a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":3087207983},{"type":"v1","addr":"172.21.15.176:6825","nonce":3087207983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":3087207983},{"type":"v1","addr":"172.21.15.176:6827","nonce":3087207983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":3087207983},{"type":"v1","addr":"172.21.15.176:6831","nonce":3087207983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":3087207983},{"type":"v1","addr":"172.21.15.176:6829","nonce":3087207983}]},"public_addr":"172.21.15.176:6825/3087207983","cluster_addr":"172.21.15.176:6827/3087207983","heartbeat_back_addr":"172.21.15.176:6831/3087207983","heartbeat_front_addr":"172.21.15.176:6829/3087207983","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:51:44.259281+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:52:10.175114+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:52:35.921506+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:53:02.271692+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:53:26.046322+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:53:51.003242+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:54:16.120316+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:54:42.320346+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/3293519534":"2024-02-11T05:50:23.907719+0000","172.21.15.64:6801/4075411303":"2024-02-11T05:50:23.907719+0000","172.21.15.64:6800/4075411303":"2024-02-11T05:50:23.907719+0000","172.21.15.64:0/1368921613":"2024-02-11T05:49:56.679796+0000","172.21.15.64:0/3226601171":"2024-02-11T05:49:56.679796+0000","172.21.15.64:0/3433552474":"2024-02-11T05:49:56.679796+0000","172.21.15.64:6800/2493684466":"2024-02-11T05:49:56.679796+0000","172.21.15.64:6801/2493684466":"2024-02-11T05:49:56.679796+0000","172.21.15.64:0/1919796165":"2024-02-11T05:50:23.907719+0000","172.21.15.64:0/77145140":"2024-02-11T05:50:23.907719+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T05:54:47.563 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-10T05:52:38.253691+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-10T05:54:47.564 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-10T05:54:47.810 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:54:47.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[25185]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:54:47.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[25185]: osdmap e57: 8 total, 8 up, 8 in 2024-02-10T05:54:47.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/3874291915' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:54:47.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:47.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:47.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:47.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:47.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:47.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:47.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:47.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:47.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:47.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:47.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:47.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:47.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:47.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:47.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[38958]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:54:47.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[38958]: osdmap e57: 8 total, 8 up, 8 in 2024-02-10T05:54:47.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/3874291915' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:54:47.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:47.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:47.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:47.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:47.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:47.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:47.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:47.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:47.879 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:47.879 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:47.879 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:47.879 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:47.879 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:47.879 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:47 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:47.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:47 smithi176 ceph-mon[30294]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:54:47.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:47 smithi176 ceph-mon[30294]: osdmap e57: 8 total, 8 up, 8 in 2024-02-10T05:54:47.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:47 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/3874291915' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:54:47.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:47 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:47.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:47 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:47.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:47 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:47.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:47 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:47.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:47 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:47.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:47 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:54:47.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:47 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:47.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:47 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:47.990 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:47 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:47.990 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:47 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:47.990 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:47 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:47.990 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:47 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:47.990 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:47 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:47.990 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:47 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:48.979 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:48 smithi064 ceph-mon[25185]: Detected new or changed devices on smithi176 2024-02-10T05:54:48.979 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:48 smithi064 ceph-mon[25185]: Adjusting osd_memory_target on smithi176 to 4272M 2024-02-10T05:54:48.979 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:48 smithi064 ceph-mon[25185]: osdmap e58: 8 total, 8 up, 8 in 2024-02-10T05:54:48.979 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:48 smithi064 ceph-mon[38958]: Detected new or changed devices on smithi176 2024-02-10T05:54:48.979 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:48 smithi064 ceph-mon[38958]: Adjusting osd_memory_target on smithi176 to 4272M 2024-02-10T05:54:48.979 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:48 smithi064 ceph-mon[38958]: osdmap e58: 8 total, 8 up, 8 in 2024-02-10T05:54:48.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:48 smithi176 ceph-mon[30294]: Detected new or changed devices on smithi176 2024-02-10T05:54:48.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:48 smithi176 ceph-mon[30294]: Adjusting osd_memory_target on smithi176 to 4272M 2024-02-10T05:54:48.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:48 smithi176 ceph-mon[30294]: osdmap e58: 8 total, 8 up, 8 in 2024-02-10T05:54:49.515 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2024-02-10T05:54:49.978 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:49 smithi064 ceph-mon[25185]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:54:49.978 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:49 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/3254693209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T05:54:49.978 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:49 smithi064 ceph-mon[38958]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:54:49.978 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:49 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/3254693209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T05:54:49.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:49 smithi176 ceph-mon[30294]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:54:49.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:49 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/3254693209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T05:54:50.273 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi064 2024-02-10T05:54:50.274 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi064=foo.a' 2024-02-10T05:54:50.524 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:54:51.957 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:51 smithi176 ceph-mon[30294]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-02-10T05:54:52.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:51 smithi064 ceph-mon[25185]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-02-10T05:54:52.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:51 smithi064 ceph-mon[38958]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-02-10T05:54:52.404 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled rgw.foo.a update... 2024-02-10T05:54:53.045 DEBUG:teuthology.orchestra.run.smithi064:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@rgw.foo.a.service 2024-02-10T05:54:53.048 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi176 2024-02-10T05:54:53.048 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-02-10T05:54:53.285 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:54:53.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[25185]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-02-10T05:54:53.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[25185]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi064=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:53.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[25185]: Saving service rgw.foo.a spec with placement smithi064=foo.a;count:1 2024-02-10T05:54:53.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:53.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:53.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:53.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:53.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:53.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T05:54:53.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T05:54:53.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:53.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:53.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[25185]: Deploying daemon rgw.foo.a on smithi064 2024-02-10T05:54:53.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[38958]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-02-10T05:54:53.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[38958]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi064=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:53.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[38958]: Saving service rgw.foo.a spec with placement smithi064=foo.a;count:1 2024-02-10T05:54:53.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:53.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:53.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:53.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:53.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:53.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T05:54:53.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T05:54:53.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:53.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:53.629 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:53 smithi064 ceph-mon[38958]: Deploying daemon rgw.foo.a on smithi064 2024-02-10T05:54:53.697 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:53 smithi176 ceph-mon[30294]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-02-10T05:54:53.697 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:53 smithi176 ceph-mon[30294]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi064=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:54:53.697 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:53 smithi176 ceph-mon[30294]: Saving service rgw.foo.a spec with placement smithi064=foo.a;count:1 2024-02-10T05:54:53.697 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:53 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:53.697 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:53 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:53.697 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:53 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:53.697 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:53 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:53.697 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:53 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:53.697 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:53 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T05:54:53.697 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:53 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T05:54:53.698 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:53 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:53.698 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:53 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:53.698 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:53 smithi176 ceph-mon[30294]: Deploying daemon rgw.foo.a on smithi064 2024-02-10T05:54:54.876 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Feb 10 05:54:54 smithi064 systemd[1]: Starting Ceph rgw.foo.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:54:55.207 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Feb 10 05:54:54 smithi064 podman[74491]: 2024-02-10 05:54:54.792964148 +0000 UTC m=+0.017566309 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:54:55.207 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Feb 10 05:54:54 smithi064 podman[74491]: 2024-02-10 05:54:54.896195882 +0000 UTC m=+0.120798036 container create 760980f4d0054d375eb4232b79fc5b66d11a485c9b443107ab0e21f71742e74b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-rgw-foo-a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-02-10T05:54:55.207 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Feb 10 05:54:55 smithi064 podman[74491]: 2024-02-10 05:54:55.1237802 +0000 UTC m=+0.348382355 container init 760980f4d0054d375eb4232b79fc5b66d11a485c9b443107ab0e21f71742e74b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-rgw-foo-a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T05:54:55.207 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Feb 10 05:54:55 smithi064 podman[74491]: 2024-02-10 05:54:55.12658346 +0000 UTC m=+0.351185623 container start 760980f4d0054d375eb4232b79fc5b66d11a485c9b443107ab0e21f71742e74b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-rgw-foo-a, RELEASE=main-290212a, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2) 2024-02-10T05:54:55.207 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Feb 10 05:54:55 smithi064 bash[74491]: 760980f4d0054d375eb4232b79fc5b66d11a485c9b443107ab0e21f71742e74b 2024-02-10T05:54:55.528 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Feb 10 05:54:55 smithi064 systemd[1]: Started Ceph rgw.foo.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:54:55.528 INFO:teuthology.orchestra.run.smithi176.stderr:pool 'datapool' created 2024-02-10T05:54:55.814 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:55 smithi176 ceph-mon[30294]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-10T05:54:55.814 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:55 smithi176 ceph-mon[30294]: from='client.? 172.21.15.176:0/2019742907' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-10T05:54:55.814 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:55 smithi176 ceph-mon[30294]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-10T05:54:55.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:55 smithi064 ceph-mon[25185]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-10T05:54:55.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:55 smithi064 ceph-mon[25185]: from='client.? 172.21.15.176:0/2019742907' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-10T05:54:55.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:55 smithi064 ceph-mon[25185]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-10T05:54:55.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:55 smithi064 ceph-mon[38958]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-10T05:54:55.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:55 smithi064 ceph-mon[38958]: from='client.? 172.21.15.176:0/2019742907' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-10T05:54:55.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:55 smithi064 ceph-mon[38958]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-10T05:54:56.050 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- rbd pool init datapool 2024-02-10T05:54:56.286 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:54:56.709 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:56 smithi176 ceph-mon[30294]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-10T05:54:56.710 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:56 smithi176 ceph-mon[30294]: osdmap e59: 8 total, 8 up, 8 in 2024-02-10T05:54:56.710 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:56 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/1982085317' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T05:54:56.710 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:56 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:56.710 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:56 smithi176 ceph-mon[30294]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T05:54:56.710 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:56 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:56.710 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:56 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:56.710 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:56 smithi176 ceph-mon[30294]: Saving service rgw.foo.a spec with placement smithi064=foo.a;count:1 2024-02-10T05:54:56.710 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:56 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:56.710 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:56 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:56.710 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:56 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:56.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:56 smithi064 ceph-mon[25185]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-10T05:54:56.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:56 smithi064 ceph-mon[25185]: osdmap e59: 8 total, 8 up, 8 in 2024-02-10T05:54:56.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:56 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/1982085317' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T05:54:56.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:56 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:56.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:56 smithi064 ceph-mon[25185]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T05:54:56.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:56 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:56.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:56 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:56.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:56 smithi064 ceph-mon[25185]: Saving service rgw.foo.a spec with placement smithi064=foo.a;count:1 2024-02-10T05:54:56.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:56 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:56.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:56 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:56.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:56 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:56.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:56 smithi064 ceph-mon[38958]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-10T05:54:56.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:56 smithi064 ceph-mon[38958]: osdmap e59: 8 total, 8 up, 8 in 2024-02-10T05:54:56.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:56 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/1982085317' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T05:54:56.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:56 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:56.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:56 smithi064 ceph-mon[38958]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T05:54:56.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:56 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:56.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:56 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:56.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:56 smithi064 ceph-mon[38958]: Saving service rgw.foo.a spec with placement smithi064=foo.a;count:1 2024-02-10T05:54:56.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:56 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:56.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:56 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:56.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:56 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:54:57.717 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:57 smithi176 ceph-mon[30294]: pgmap v185: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-02-10T05:54:57.718 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:57 smithi176 ceph-mon[30294]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-10T05:54:57.718 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:57 smithi176 ceph-mon[30294]: osdmap e60: 8 total, 8 up, 8 in 2024-02-10T05:54:57.718 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:57 smithi176 ceph-mon[30294]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T05:54:57.718 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:57 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:57.718 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:57 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:57.718 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:57 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:57.718 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:57 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:57.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:57 smithi064 ceph-mon[25185]: pgmap v185: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-02-10T05:54:57.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:57 smithi064 ceph-mon[25185]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-10T05:54:57.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:57 smithi064 ceph-mon[25185]: osdmap e60: 8 total, 8 up, 8 in 2024-02-10T05:54:57.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:57 smithi064 ceph-mon[25185]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T05:54:57.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:57 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:57.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:57 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:57.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:57 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:57.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:57 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:57.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:57 smithi064 ceph-mon[38958]: pgmap v185: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-02-10T05:54:57.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:57 smithi064 ceph-mon[38958]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-10T05:54:57.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:57 smithi064 ceph-mon[38958]: osdmap e60: 8 total, 8 up, 8 in 2024-02-10T05:54:57.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:57 smithi064 ceph-mon[38958]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-10T05:54:57.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:57 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:57.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:57 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:57.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:57 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:54:57.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:57 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:54:58.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:58 smithi064 ceph-mon[25185]: Checking dashboard <-> RGW credentials 2024-02-10T05:54:58.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:58 smithi064 ceph-mon[25185]: osdmap e61: 8 total, 8 up, 8 in 2024-02-10T05:54:58.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:58 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/2087883926' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:54:58.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:58 smithi064 ceph-mon[25185]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:54:58.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:58 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/2413413130' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:54:58.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:58 smithi064 ceph-mon[25185]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:54:58.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:58 smithi064 ceph-mon[25185]: from='client.? 172.21.15.176:0/2463421001' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-10T05:54:58.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:58 smithi064 ceph-mon[25185]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-10T05:54:58.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:58 smithi064 ceph-mon[38958]: Checking dashboard <-> RGW credentials 2024-02-10T05:54:58.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:58 smithi064 ceph-mon[38958]: osdmap e61: 8 total, 8 up, 8 in 2024-02-10T05:54:58.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:58 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/2087883926' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:54:58.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:58 smithi064 ceph-mon[38958]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:54:58.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:58 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/2413413130' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:54:58.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:58 smithi064 ceph-mon[38958]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:54:58.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:58 smithi064 ceph-mon[38958]: from='client.? 172.21.15.176:0/2463421001' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-10T05:54:58.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:58 smithi064 ceph-mon[38958]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-10T05:54:58.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:58 smithi176 ceph-mon[30294]: Checking dashboard <-> RGW credentials 2024-02-10T05:54:58.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:58 smithi176 ceph-mon[30294]: osdmap e61: 8 total, 8 up, 8 in 2024-02-10T05:54:58.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:58 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/2087883926' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:54:58.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:58 smithi176 ceph-mon[30294]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:54:58.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:58 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/2413413130' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:54:58.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:58 smithi176 ceph-mon[30294]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:54:58.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:58 smithi176 ceph-mon[30294]: from='client.? 172.21.15.176:0/2463421001' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-10T05:54:58.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:58 smithi176 ceph-mon[30294]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-10T05:54:59.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:59 smithi064 ceph-mon[25185]: pgmap v188: 68 pgs: 7 creating+peering, 16 active+clean, 45 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-02-10T05:54:59.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:59 smithi064 ceph-mon[25185]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T05:54:59.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:59 smithi064 ceph-mon[25185]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T05:54:59.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:59 smithi064 ceph-mon[25185]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-02-10T05:54:59.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:59 smithi064 ceph-mon[25185]: osdmap e62: 8 total, 8 up, 8 in 2024-02-10T05:54:59.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:54:59 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:59.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:59 smithi064 ceph-mon[38958]: pgmap v188: 68 pgs: 7 creating+peering, 16 active+clean, 45 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-02-10T05:54:59.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:59 smithi064 ceph-mon[38958]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T05:54:59.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:59 smithi064 ceph-mon[38958]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T05:54:59.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:59 smithi064 ceph-mon[38958]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-02-10T05:54:59.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:59 smithi064 ceph-mon[38958]: osdmap e62: 8 total, 8 up, 8 in 2024-02-10T05:54:59.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:54:59 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:54:59.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:59 smithi176 ceph-mon[30294]: pgmap v188: 68 pgs: 7 creating+peering, 16 active+clean, 45 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-02-10T05:54:59.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:59 smithi176 ceph-mon[30294]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T05:54:59.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:59 smithi176 ceph-mon[30294]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T05:54:59.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:59 smithi176 ceph-mon[30294]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-02-10T05:54:59.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:59 smithi176 ceph-mon[30294]: osdmap e62: 8 total, 8 up, 8 in 2024-02-10T05:54:59.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:54:59 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:00.853 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:00 smithi176 ceph-mon[30294]: osdmap e63: 8 total, 8 up, 8 in 2024-02-10T05:55:00.853 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:00 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/2087883926' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:55:00.853 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:00 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/2413413130' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:55:00.853 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:00 smithi176 ceph-mon[30294]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:55:00.853 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:00 smithi176 ceph-mon[30294]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:55:00.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:00 smithi064 ceph-mon[25185]: osdmap e63: 8 total, 8 up, 8 in 2024-02-10T05:55:00.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:00 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/2087883926' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:55:00.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:00 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/2413413130' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:55:00.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:00 smithi064 ceph-mon[25185]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:55:00.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:00 smithi064 ceph-mon[25185]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:55:00.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:00 smithi064 ceph-mon[38958]: osdmap e63: 8 total, 8 up, 8 in 2024-02-10T05:55:00.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:00 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/2087883926' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:55:00.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:00 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/2413413130' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:55:00.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:00 smithi064 ceph-mon[38958]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:55:00.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:00 smithi064 ceph-mon[38958]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:55:01.162 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.176 --placement '1;smithi176=iscsi.a' 2024-02-10T05:55:01.402 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:55:01.823 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:01 smithi176 ceph-mon[30294]: pgmap v191: 100 pgs: 14 creating+peering, 40 active+clean, 46 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-10T05:55:01.824 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:01 smithi176 ceph-mon[30294]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T05:55:01.824 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:01 smithi176 ceph-mon[30294]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T05:55:01.824 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:01 smithi176 ceph-mon[30294]: osdmap e64: 8 total, 8 up, 8 in 2024-02-10T05:55:01.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:01 smithi064 ceph-mon[25185]: pgmap v191: 100 pgs: 14 creating+peering, 40 active+clean, 46 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-10T05:55:01.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:01 smithi064 ceph-mon[25185]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T05:55:01.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:01 smithi064 ceph-mon[25185]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T05:55:01.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:01 smithi064 ceph-mon[25185]: osdmap e64: 8 total, 8 up, 8 in 2024-02-10T05:55:01.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:01 smithi064 ceph-mon[38958]: pgmap v191: 100 pgs: 14 creating+peering, 40 active+clean, 46 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-10T05:55:01.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:01 smithi064 ceph-mon[38958]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T05:55:01.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:01 smithi064 ceph-mon[38958]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T05:55:01.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:01 smithi064 ceph-mon[38958]: osdmap e64: 8 total, 8 up, 8 in 2024-02-10T05:55:02.699 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:02 smithi176 ceph-mon[30294]: osdmap e65: 8 total, 8 up, 8 in 2024-02-10T05:55:02.699 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:02 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/2087883926' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:55:02.699 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:02 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/2413413130' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:55:02.699 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:02 smithi176 ceph-mon[30294]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:55:02.699 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:02 smithi176 ceph-mon[30294]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:55:02.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:02 smithi064 ceph-mon[25185]: osdmap e65: 8 total, 8 up, 8 in 2024-02-10T05:55:02.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:02 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/2087883926' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:55:02.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:02 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/2413413130' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:55:02.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:02 smithi064 ceph-mon[25185]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:55:02.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:02 smithi064 ceph-mon[25185]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:55:02.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:02 smithi064 ceph-mon[38958]: osdmap e65: 8 total, 8 up, 8 in 2024-02-10T05:55:02.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:02 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/2087883926' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:55:02.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:02 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/2413413130' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:55:02.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:02 smithi064 ceph-mon[38958]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:55:02.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:02 smithi064 ceph-mon[38958]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:55:03.405 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled iscsi.datapool update... 2024-02-10T05:55:03.738 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:03 smithi176 ceph-mon[30294]: pgmap v194: 132 pgs: 23 creating+peering, 61 active+clean, 48 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-02-10T05:55:03.738 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:03 smithi176 ceph-mon[30294]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T05:55:03.738 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:03 smithi176 ceph-mon[30294]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T05:55:03.738 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:03 smithi176 ceph-mon[30294]: osdmap e66: 8 total, 8 up, 8 in 2024-02-10T05:55:03.738 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:03 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/2087883926' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:55:03.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:03 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/2413413130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:55:03.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:03 smithi176 ceph-mon[30294]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:55:03.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:03 smithi176 ceph-mon[30294]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:55:03.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:03 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:03.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:03 smithi064 ceph-mon[25185]: pgmap v194: 132 pgs: 23 creating+peering, 61 active+clean, 48 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-02-10T05:55:03.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:03 smithi064 ceph-mon[25185]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T05:55:03.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:03 smithi064 ceph-mon[25185]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T05:55:03.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:03 smithi064 ceph-mon[25185]: osdmap e66: 8 total, 8 up, 8 in 2024-02-10T05:55:03.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:03 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/2087883926' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:55:03.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:03 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/2413413130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:55:03.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:03 smithi064 ceph-mon[25185]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:55:03.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:03 smithi064 ceph-mon[25185]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:55:03.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:03 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:03.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:03 smithi064 ceph-mon[38958]: pgmap v194: 132 pgs: 23 creating+peering, 61 active+clean, 48 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-02-10T05:55:03.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:03 smithi064 ceph-mon[38958]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T05:55:03.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:03 smithi064 ceph-mon[38958]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T05:55:03.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:03 smithi064 ceph-mon[38958]: osdmap e66: 8 total, 8 up, 8 in 2024-02-10T05:55:03.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:03 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/2087883926' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:55:03.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:03 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/2413413130' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:55:03.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:03 smithi064 ceph-mon[38958]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:55:03.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:03 smithi064 ceph-mon[38958]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:55:03.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:03 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:03.994 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-02-10T05:55:03.994 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T05:55:03.994 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-02-10T05:55:04.025 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-10T05:55:04.025 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-02-10T05:55:04.055 DEBUG:teuthology.orchestra.run.smithi176:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@iscsi.iscsi.a.service 2024-02-10T05:55:04.097 INFO:tasks.cephadm:Adding prometheus.a on smithi176 2024-02-10T05:55:04.097 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph orch apply prometheus '1;smithi176=a' 2024-02-10T05:55:04.378 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:55:04.824 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:04 smithi176 ceph-mon[30294]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.176", "placement": "1;smithi176=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:04.825 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:04 smithi176 ceph-mon[30294]: Saving service iscsi.datapool spec with placement smithi176=iscsi.a;count:1 2024-02-10T05:55:04.825 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:04 smithi176 ceph-mon[30294]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T05:55:04.825 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:04 smithi176 ceph-mon[30294]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T05:55:04.825 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:04 smithi176 ceph-mon[30294]: osdmap e67: 8 total, 8 up, 8 in 2024-02-10T05:55:04.825 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:04 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:04.825 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:04 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:04.825 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:04 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:04.825 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:04 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T05:55:04.825 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:04 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-10T05:55:04.825 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:04 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:04.825 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:04 smithi176 ceph-mon[30294]: Deploying daemon iscsi.iscsi.a on smithi176 2024-02-10T05:55:04.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:04 smithi064 ceph-mon[25185]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.176", "placement": "1;smithi176=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:04.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:04 smithi064 ceph-mon[25185]: Saving service iscsi.datapool spec with placement smithi176=iscsi.a;count:1 2024-02-10T05:55:04.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:04 smithi064 ceph-mon[25185]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T05:55:04.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:04 smithi064 ceph-mon[25185]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T05:55:04.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:04 smithi064 ceph-mon[25185]: osdmap e67: 8 total, 8 up, 8 in 2024-02-10T05:55:04.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:04 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:04.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:04 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:04.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:04 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:04.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:04 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T05:55:04.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:04 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-10T05:55:04.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:04 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:04.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:04 smithi064 ceph-mon[25185]: Deploying daemon iscsi.iscsi.a on smithi176 2024-02-10T05:55:04.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:04 smithi064 ceph-mon[38958]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.176", "placement": "1;smithi176=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:04.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:04 smithi064 ceph-mon[38958]: Saving service iscsi.datapool spec with placement smithi176=iscsi.a;count:1 2024-02-10T05:55:04.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:04 smithi064 ceph-mon[38958]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T05:55:04.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:04 smithi064 ceph-mon[38958]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T05:55:04.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:04 smithi064 ceph-mon[38958]: osdmap e67: 8 total, 8 up, 8 in 2024-02-10T05:55:04.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:04 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:04.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:04 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:04.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:04 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:04.879 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:04 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-10T05:55:04.879 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:04 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-10T05:55:04.879 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:04 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:04.879 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:04 smithi064 ceph-mon[38958]: Deploying daemon iscsi.iscsi.a on smithi176 2024-02-10T05:55:05.854 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:05 smithi176 ceph-mon[30294]: pgmap v197: 132 pgs: 2 creating+activating, 13 creating+peering, 95 active+clean, 22 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-02-10T05:55:05.854 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:05 smithi176 ceph-mon[30294]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T05:55:05.855 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:05 smithi176 ceph-mon[30294]: Cluster is now healthy 2024-02-10T05:55:05.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:05 smithi064 ceph-mon[25185]: pgmap v197: 132 pgs: 2 creating+activating, 13 creating+peering, 95 active+clean, 22 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-02-10T05:55:05.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:05 smithi064 ceph-mon[25185]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T05:55:05.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:05 smithi064 ceph-mon[25185]: Cluster is now healthy 2024-02-10T05:55:05.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:05 smithi064 ceph-mon[38958]: pgmap v197: 132 pgs: 2 creating+activating, 13 creating+peering, 95 active+clean, 22 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-02-10T05:55:05.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:05 smithi064 ceph-mon[38958]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-10T05:55:05.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:05 smithi064 ceph-mon[38958]: Cluster is now healthy 2024-02-10T05:55:07.091 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled prometheus update... 2024-02-10T05:55:07.699 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:07 smithi176 ceph-mon[30294]: pgmap v198: 132 pgs: 2 creating+activating, 5 creating+peering, 125 active+clean; 580 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 3.9 KiB/s wr, 155 op/s 2024-02-10T05:55:07.699 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:07 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:07.725 DEBUG:teuthology.orchestra.run.smithi176:prometheus.a> sudo journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@prometheus.a.service 2024-02-10T05:55:07.727 INFO:tasks.cephadm:Adding node-exporter.a on smithi064 2024-02-10T05:55:07.727 INFO:tasks.cephadm:Adding node-exporter.b on smithi176 2024-02-10T05:55:07.727 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph orch apply node-exporter '2;smithi064=a;smithi176=b' 2024-02-10T05:55:07.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:07 smithi064 ceph-mon[25185]: pgmap v198: 132 pgs: 2 creating+activating, 5 creating+peering, 125 active+clean; 580 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 3.9 KiB/s wr, 155 op/s 2024-02-10T05:55:07.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:07 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:07.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:07 smithi064 ceph-mon[38958]: pgmap v198: 132 pgs: 2 creating+activating, 5 creating+peering, 125 active+clean; 580 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 3.9 KiB/s wr, 155 op/s 2024-02-10T05:55:07.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:07 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:07.949 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:55:07 smithi176 podman[66223]: 2024-02-10 05:55:07.839250558 +0000 UTC m=+0.018014412 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:55:07.949 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:55:07 smithi176 podman[66223]: 2024-02-10 05:55:07.949746227 +0000 UTC m=+0.128510076 container create f7e7f53e917618d3625b11bda0634b4bc70acc1abd87090bec727b261a874015 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:55:08.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:55:08 smithi176 podman[66223]: 2024-02-10 05:55:08.160297881 +0000 UTC m=+0.339061727 container init f7e7f53e917618d3625b11bda0634b4bc70acc1abd87090bec727b261a874015 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-02-10T05:55:08.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:55:08 smithi176 podman[66223]: 2024-02-10 05:55:08.163672854 +0000 UTC m=+0.342436701 container start f7e7f53e917618d3625b11bda0634b4bc70acc1abd87090bec727b261a874015 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:55:08.340 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:55:08.609 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:55:08 smithi176 bash[66223]: f7e7f53e917618d3625b11bda0634b4bc70acc1abd87090bec727b261a874015 2024-02-10T05:55:08.609 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:55:08 smithi176 systemd[1]: Started Ceph iscsi.iscsi.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:55:08.609 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:55:08 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[66249]: debug Started the configuration object watcher 2024-02-10T05:55:08.609 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:55:08 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[66249]: debug Checking for config object changes every 1s 2024-02-10T05:55:08.609 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:55:08 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[66249]: debug Processing osd blocklist entries for this node 2024-02-10T05:55:08.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:08 smithi064 ceph-mon[25185]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi176=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:08.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:08 smithi064 ceph-mon[25185]: Saving service prometheus spec with placement smithi176=a;count:1 2024-02-10T05:55:08.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:08 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:08.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:08 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:08.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:08 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:08.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:08 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:08.877 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:08 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:08.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:08 smithi064 ceph-mon[38958]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi176=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:08.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:08 smithi064 ceph-mon[38958]: Saving service prometheus spec with placement smithi176=a;count:1 2024-02-10T05:55:08.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:08 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:08.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:08 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:08.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:08 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:08.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:08 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:08.878 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:08 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:08.916 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:55:08 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[66249]: debug Reading the configuration object to update local LIO configuration 2024-02-10T05:55:08.916 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:55:08 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[66249]: debug Configuration does not have an entry for this host(smithi176.front.sepia.ceph.com) - nothing to define to LIO 2024-02-10T05:55:08.916 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:55:08 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[66249]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-02-10T05:55:08.916 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:08 smithi176 ceph-mon[30294]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi176=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:08.916 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:08 smithi176 ceph-mon[30294]: Saving service prometheus spec with placement smithi176=a;count:1 2024-02-10T05:55:08.916 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:08 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:08.917 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:08 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:08.917 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:08 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:08.917 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:08 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:08.917 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:08 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:09.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:09 smithi064 ceph-mon[25185]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 5.6 KiB/s wr, 227 op/s 2024-02-10T05:55:09.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:09 smithi064 ceph-mon[25185]: Checking pool "datapool" exists for service iscsi.datapool 2024-02-10T05:55:09.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:09 smithi064 ceph-mon[25185]: from='client.? 172.21.15.176:0/2806244018' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T05:55:09.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:09 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:09.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:09 smithi064 ceph-mon[38958]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 5.6 KiB/s wr, 227 op/s 2024-02-10T05:55:09.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:09 smithi064 ceph-mon[38958]: Checking pool "datapool" exists for service iscsi.datapool 2024-02-10T05:55:09.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:09 smithi064 ceph-mon[38958]: from='client.? 172.21.15.176:0/2806244018' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T05:55:09.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:09 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:09.948 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:09 smithi176 ceph-mon[30294]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 5.6 KiB/s wr, 227 op/s 2024-02-10T05:55:09.948 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:09 smithi176 ceph-mon[30294]: Checking pool "datapool" exists for service iscsi.datapool 2024-02-10T05:55:09.948 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:09 smithi176 ceph-mon[30294]: from='client.? 172.21.15.176:0/2806244018' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-10T05:55:09.948 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:09 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:10.513 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled node-exporter update... 2024-02-10T05:55:11.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:11 smithi064 ceph-mon[25185]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 4.9 KiB/s wr, 202 op/s 2024-02-10T05:55:11.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:11 smithi064 ceph-mon[25185]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-10T05:55:11.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:11 smithi064 ceph-mon[25185]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi064=a;smithi176=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:11.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:11 smithi064 ceph-mon[25185]: Saving service node-exporter spec with placement smithi064=a;smithi176=b;count:2 2024-02-10T05:55:11.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:11 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:11.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:11 smithi064 ceph-mon[25185]: osdmap e68: 8 total, 8 up, 8 in 2024-02-10T05:55:11.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:11 smithi064 ceph-mon[38958]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 4.9 KiB/s wr, 202 op/s 2024-02-10T05:55:11.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:11 smithi064 ceph-mon[38958]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-10T05:55:11.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:11 smithi064 ceph-mon[38958]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi064=a;smithi176=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:11.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:11 smithi064 ceph-mon[38958]: Saving service node-exporter spec with placement smithi064=a;smithi176=b;count:2 2024-02-10T05:55:11.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:11 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:11.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:11 smithi064 ceph-mon[38958]: osdmap e68: 8 total, 8 up, 8 in 2024-02-10T05:55:11.382 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:11 smithi176 ceph-mon[30294]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 4.9 KiB/s wr, 202 op/s 2024-02-10T05:55:11.382 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:11 smithi176 ceph-mon[30294]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-10T05:55:11.382 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:11 smithi176 ceph-mon[30294]: from='client.24424 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi064=a;smithi176=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:11.382 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:11 smithi176 ceph-mon[30294]: Saving service node-exporter spec with placement smithi064=a;smithi176=b;count:2 2024-02-10T05:55:11.382 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:11 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:11.382 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:11 smithi176 ceph-mon[30294]: osdmap e68: 8 total, 8 up, 8 in 2024-02-10T05:55:11.415 DEBUG:teuthology.orchestra.run.smithi064:node-exporter.a> sudo journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@node-exporter.a.service 2024-02-10T05:55:11.418 DEBUG:teuthology.orchestra.run.smithi176:node-exporter.b> sudo journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@node-exporter.b.service 2024-02-10T05:55:11.420 INFO:tasks.cephadm:Adding alertmanager.a on smithi064 2024-02-10T05:55:11.420 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph orch apply alertmanager '1;smithi064=a' 2024-02-10T05:55:12.349 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:55:13.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:13 smithi064 ceph-mon[25185]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 4.2 KiB/s wr, 191 op/s 2024-02-10T05:55:13.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:13 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:13.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:13 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:13.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:13 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:13.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:13 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:13.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:13 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:13.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:13 smithi064 ceph-mon[25185]: Deploying daemon prometheus.a on smithi176 2024-02-10T05:55:13.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:13 smithi064 ceph-mon[38958]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 4.2 KiB/s wr, 191 op/s 2024-02-10T05:55:13.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:13 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:13.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:13 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:13.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:13 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:13.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:13 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:13.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:13 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:13.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:13 smithi064 ceph-mon[38958]: Deploying daemon prometheus.a on smithi176 2024-02-10T05:55:13.709 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:13 smithi176 ceph-mon[30294]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 4.2 KiB/s wr, 191 op/s 2024-02-10T05:55:13.709 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:13 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:13.709 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:13 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:13.709 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:13 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:55:13.710 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:13 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:55:13.710 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:13 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:13.710 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:13 smithi176 ceph-mon[30294]: Deploying daemon prometheus.a on smithi176 2024-02-10T05:55:14.298 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled alertmanager update... 2024-02-10T05:55:15.136 DEBUG:teuthology.orchestra.run.smithi064:alertmanager.a> sudo journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@alertmanager.a.service 2024-02-10T05:55:15.139 INFO:tasks.cephadm:Adding grafana.a on smithi176 2024-02-10T05:55:15.139 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph orch apply grafana '1;smithi176=a' 2024-02-10T05:55:15.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:15 smithi064 ceph-mon[25185]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 3.4 KiB/s wr, 157 op/s 2024-02-10T05:55:15.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:15 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:15.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:15 smithi064 ceph-mon[25185]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi064=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:15.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:15 smithi064 ceph-mon[25185]: Saving service alertmanager spec with placement smithi064=a;count:1 2024-02-10T05:55:15.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:15 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:15.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:15 smithi064 ceph-mon[38958]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 3.4 KiB/s wr, 157 op/s 2024-02-10T05:55:15.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:15 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:15.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:15 smithi064 ceph-mon[38958]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi064=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:15.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:15 smithi064 ceph-mon[38958]: Saving service alertmanager spec with placement smithi064=a;count:1 2024-02-10T05:55:15.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:15 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:15.381 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:55:15.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:15 smithi176 ceph-mon[30294]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s rd, 3.4 KiB/s wr, 157 op/s 2024-02-10T05:55:15.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:15 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:15.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:15 smithi176 ceph-mon[30294]: from='client.24430 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi064=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:15.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:15 smithi176 ceph-mon[30294]: Saving service alertmanager spec with placement smithi064=a;count:1 2024-02-10T05:55:15.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:15 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:17.638 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:17 smithi176 ceph-mon[30294]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 1.8 KiB/s wr, 78 op/s 2024-02-10T05:55:17.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:17 smithi064 ceph-mon[25185]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 1.8 KiB/s wr, 78 op/s 2024-02-10T05:55:17.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:17 smithi064 ceph-mon[38958]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 1.8 KiB/s wr, 78 op/s 2024-02-10T05:55:17.888 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled grafana update... 2024-02-10T05:55:18.738 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:55:18 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[66249]: debug there is no tcmu-runner data available 2024-02-10T05:55:19.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:18 smithi176 ceph-mon[30294]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi176=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:19.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:18 smithi176 ceph-mon[30294]: Saving service grafana spec with placement smithi176=a;count:1 2024-02-10T05:55:19.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:18 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:19.270 DEBUG:teuthology.orchestra.run.smithi176:grafana.a> sudo journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@grafana.a.service 2024-02-10T05:55:19.272 INFO:tasks.cephadm:Setting up client nodes... 2024-02-10T05:55:19.273 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-10T05:55:19.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:18 smithi064 ceph-mon[25185]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi176=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:19.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:18 smithi064 ceph-mon[25185]: Saving service grafana spec with placement smithi176=a;count:1 2024-02-10T05:55:19.301 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:18 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:19.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:18 smithi064 ceph-mon[38958]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi176=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:55:19.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:18 smithi064 ceph-mon[38958]: Saving service grafana spec with placement smithi176=a;count:1 2024-02-10T05:55:19.301 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:18 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:19.529 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:55:20.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:19 smithi064 ceph-mon[38958]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 307 B/s wr, 17 op/s 2024-02-10T05:55:20.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:19 smithi064 ceph-mon[38958]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:55:20.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:19 smithi064 ceph-mon[25185]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 307 B/s wr, 17 op/s 2024-02-10T05:55:20.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:19 smithi064 ceph-mon[25185]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:55:20.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:19 smithi176 ceph-mon[30294]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 307 B/s wr, 17 op/s 2024-02-10T05:55:20.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:19 smithi176 ceph-mon[30294]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:55:21.587 INFO:teuthology.orchestra.run.smithi064.stdout:[client.0] 2024-02-10T05:55:21.587 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQDJD8dl1T/PIhAAKqu4uj2eXV/p3pQz9Y0Hug== 2024-02-10T05:55:22.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:21 smithi064 ceph-mon[25185]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T05:55:22.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:21 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/1718038126' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:55:22.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:21 smithi064 ceph-mon[25185]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:55:22.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:21 smithi064 ceph-mon[25185]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T05:55:22.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:21 smithi064 ceph-mon[38958]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T05:55:22.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:21 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/1718038126' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:55:22.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:21 smithi064 ceph-mon[38958]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:55:22.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:21 smithi064 ceph-mon[38958]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T05:55:22.160 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:21 smithi176 ceph-mon[30294]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-10T05:55:22.160 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:21 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/1718038126' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:55:22.160 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:21 smithi176 ceph-mon[30294]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:55:22.160 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:21 smithi176 ceph-mon[30294]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T05:55:22.181 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T05:55:22.181 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-10T05:55:22.182 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-10T05:55:22.265 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-10T05:55:23.333 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:55:24.166 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:23 smithi176 ceph-mon[30294]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 901 B/s rd, 0 op/s 2024-02-10T05:55:24.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:23 smithi064 ceph-mon[25185]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 901 B/s rd, 0 op/s 2024-02-10T05:55:24.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:23 smithi064 ceph-mon[38958]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 901 B/s rd, 0 op/s 2024-02-10T05:55:25.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:24 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-02-10T05:55:25.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:24 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-02-10T05:55:25.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:24 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-02-10T05:55:25.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:24 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-02-10T05:55:25.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:24 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-02-10T05:55:25.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:24 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-02-10T05:55:25.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:24 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-02-10T05:55:25.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:24 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-02-10T05:55:25.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:24 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-02-10T05:55:26.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:25 smithi176 ceph-mon[30294]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:55:26.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:25 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-02-10T05:55:26.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:25 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-02-10T05:55:26.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:25 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]': finished 2024-02-10T05:55:26.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:25 smithi176 ceph-mon[30294]: osdmap e69: 8 total, 8 up, 8 in 2024-02-10T05:55:26.302 INFO:teuthology.orchestra.run.smithi176.stdout:[client.1] 2024-02-10T05:55:26.302 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDOD8dlEOK4ERAABluSr05quxNlyiFWqg5XZQ== 2024-02-10T05:55:26.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:25 smithi064 ceph-mon[25185]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:55:26.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:25 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-02-10T05:55:26.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:25 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-02-10T05:55:26.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:25 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]': finished 2024-02-10T05:55:26.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:25 smithi064 ceph-mon[25185]: osdmap e69: 8 total, 8 up, 8 in 2024-02-10T05:55:26.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:25 smithi064 ceph-mon[38958]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:55:26.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:25 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-02-10T05:55:26.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:25 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-02-10T05:55:26.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:25 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]': finished 2024-02-10T05:55:26.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:25 smithi064 ceph-mon[38958]: osdmap e69: 8 total, 8 up, 8 in 2024-02-10T05:55:26.886 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-10T05:55:26.887 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-10T05:55:26.887 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-10T05:55:26.935 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-10T05:55:26.935 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-10T05:55:26.935 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph mgr dump --format=json 2024-02-10T05:55:27.165 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:26 smithi176 ceph-mon[30294]: osdmap e70: 8 total, 8 up, 8 in 2024-02-10T05:55:27.165 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:26 smithi176 ceph-mon[30294]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:55:27.165 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:26 smithi176 ceph-mon[30294]: from='client.? 172.21.15.176:0/3520365924' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:55:27.166 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:26 smithi176 ceph-mon[30294]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:55:27.166 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:26 smithi176 ceph-mon[30294]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T05:55:27.185 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:55:27.306 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:26 smithi064 ceph-mon[25185]: osdmap e70: 8 total, 8 up, 8 in 2024-02-10T05:55:27.307 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:26 smithi064 ceph-mon[25185]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:55:27.307 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:26 smithi064 ceph-mon[25185]: from='client.? 172.21.15.176:0/3520365924' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:55:27.307 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:26 smithi064 ceph-mon[25185]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:55:27.307 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:26 smithi064 ceph-mon[25185]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T05:55:27.307 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:26 smithi064 ceph-mon[38958]: osdmap e70: 8 total, 8 up, 8 in 2024-02-10T05:55:27.307 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:26 smithi064 ceph-mon[38958]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:55:27.307 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:26 smithi064 ceph-mon[38958]: from='client.? 172.21.15.176:0/3520365924' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:55:27.307 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:26 smithi064 ceph-mon[38958]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:55:27.308 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:26 smithi064 ceph-mon[38958]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T05:55:27.738 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Feb 10 05:55:27 smithi176 systemd[1]: Started Ceph prometheus.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:55:28.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:28 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:28.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:28 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:28.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:28 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:28.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:28 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T05:55:28.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:28 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:28.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:28 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:28.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:28 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:28.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:28 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T05:55:28.627 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:28 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: ignoring --setuser ceph since I am not root 2024-02-10T05:55:28.627 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:28 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: ignoring --setgroup ceph since I am not root 2024-02-10T05:55:28.738 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:28 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:28.738 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:28 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:28.738 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:28 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' 2024-02-10T05:55:28.738 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:28 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T05:55:28.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:55:28 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[66249]: debug there is no tcmu-runner data available 2024-02-10T05:55:28.739 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:28 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: ignoring --setuser ceph since I am not root 2024-02-10T05:55:28.739 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:28 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: ignoring --setgroup ceph since I am not root 2024-02-10T05:55:28.739 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:28 smithi176 ceph-mgr[32155]: -- 172.21.15.176:0/999278230 <== mon.1 v2:172.21.15.176:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560fb3adc1a0 con 0x560fb3ab3000 2024-02-10T05:55:29.109 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:55:29.376 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:29 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:29.075+0000 7f680ee5c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:55:29.376 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:29 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:29.175+0000 7f680ee5c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:55:29.483 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:29 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:29.049+0000 7fe801586200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:55:29.483 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:29 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:29.142+0000 7fe801586200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:55:29.738 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:29 smithi176 ceph-mon[30294]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T05:55:29.738 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:29 smithi176 ceph-mon[30294]: mgrmap e17: y(active, since 5m), standbys: x 2024-02-10T05:55:29.738 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:29 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/2073956868' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T05:55:29.790 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:29 smithi064 ceph-mon[25185]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T05:55:29.790 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:29 smithi064 ceph-mon[25185]: mgrmap e17: y(active, since 5m), standbys: x 2024-02-10T05:55:29.790 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:29 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/2073956868' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T05:55:29.790 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:29 smithi064 ceph-mon[38958]: from='mgr.14150 172.21.15.64:0/3981145885' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T05:55:29.790 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:29 smithi064 ceph-mon[38958]: mgrmap e17: y(active, since 5m), standbys: x 2024-02-10T05:55:29.790 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:29 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/2073956868' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T05:55:29.817 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":1905475246},{"type":"v1","addr":"172.21.15.64:6801","nonce":1905475246}]},"active_addr":"172.21.15.64:6801/1905475246","active_change":"2024-02-10T05:50:23.907881+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.64:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":2264629492}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":228286744}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":4079711029}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":1517623801}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":452158396}]}]} 2024-02-10T05:55:29.825 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-10T05:55:29.825 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-10T05:55:29.825 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T05:55:30.073 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:55:30.107 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:29 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:29.850+0000 7f680ee5c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:55:30.107 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:29 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:29.948+0000 7f680ee5c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:55:30.107 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:30 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:30.105+0000 7f680ee5c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:55:30.111 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:29 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:29.780+0000 7fe801586200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:55:30.111 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:29 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:29.872+0000 7fe801586200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:55:30.111 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:30 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:30.015+0000 7fe801586200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:55:30.111 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:30 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:30.110+0000 7fe801586200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:55:30.376 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:30 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:30.204+0000 7f680ee5c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:55:30.376 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:30 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:30.291+0000 7f680ee5c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:55:30.459 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:30 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:30.194+0000 7fe801586200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:55:30.459 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:30 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:30.458+0000 7fe801586200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T05:55:30.738 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:30 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:30.571+0000 7fe801586200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:55:30.738 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:30 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:30.658+0000 7fe801586200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:55:30.772 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:30 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:30.567+0000 7f680ee5c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T05:55:30.782 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:30 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:30.682+0000 7f680ee5c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:55:30.782 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:30 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:30.770+0000 7f680ee5c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:55:31.051 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:30 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:30.869+0000 7f680ee5c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:55:31.238 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:30 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:30.753+0000 7fe801586200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:55:31.238 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:30 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:30.931+0000 7fe801586200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:55:31.376 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:31 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:31.049+0000 7f680ee5c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:55:31.989 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:55:31.989 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":70,"fsid":"00e86dbe-c7d8-11ee-95b8-87774f69a715","created":"2024-02-10T05:49:26.002096+0000","modified":"2024-02-10T05:55:25.902709+0000","last_up_change":"2024-02-10T05:54:43.610359+0000","last_in_change":"2024-02-10T05:54:25.763533+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T05:52:38.253691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-02-10T05:54:55.269909+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-02-10T05:54:55.517233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-10T05:54:56.852965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-10T05:54:58.816169+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-10T05:55:00.886438+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"32df1361-a86f-44aa-bc33-d0b1e3fc140b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3211593074},{"type":"v1","addr":"172.21.15.64:6803","nonce":3211593074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3211593074},{"type":"v1","addr":"172.21.15.64:6805","nonce":3211593074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3211593074},{"type":"v1","addr":"172.21.15.64:6809","nonce":3211593074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3211593074},{"type":"v1","addr":"172.21.15.64:6807","nonce":3211593074}]},"public_addr":"172.21.15.64:6803/3211593074","cluster_addr":"172.21.15.64:6805/3211593074","heartbeat_back_addr":"172.21.15.64:6809/3211593074","heartbeat_front_addr":"172.21.15.64:6807/3211593074","state":["exists","up"]},{"osd":1,"uuid":"cfe981c9-406d-4da9-89ff-81dbbef91183","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":614169574},{"type":"v1","addr":"172.21.15.64:6811","nonce":614169574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":614169574},{"type":"v1","addr":"172.21.15.64:6813","nonce":614169574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":614169574},{"type":"v1","addr":"172.21.15.64:6817","nonce":614169574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":614169574},{"type":"v1","addr":"172.21.15.64:6815","nonce":614169574}]},"public_addr":"172.21.15.64:6811/614169574","cluster_addr":"172.21.15.64:6813/614169574","heartbeat_back_addr":"172.21.15.64:6817/614169574","heartbeat_front_addr":"172.21.15.64:6815/614169574","state":["exists","up"]},{"osd":2,"uuid":"bb433070-a584-427e-bb43-bf20be79b123","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2362209772},{"type":"v1","addr":"172.21.15.64:6819","nonce":2362209772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2362209772},{"type":"v1","addr":"172.21.15.64:6821","nonce":2362209772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":2362209772},{"type":"v1","addr":"172.21.15.64:6825","nonce":2362209772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2362209772},{"type":"v1","addr":"172.21.15.64:6823","nonce":2362209772}]},"public_addr":"172.21.15.64:6819/2362209772","cluster_addr":"172.21.15.64:6821/2362209772","heartbeat_back_addr":"172.21.15.64:6825/2362209772","heartbeat_front_addr":"172.21.15.64:6823/2362209772","state":["exists","up"]},{"osd":3,"uuid":"bb31284d-6c1a-4f79-928a-bb758b45e196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":256208898},{"type":"v1","addr":"172.21.15.64:6827","nonce":256208898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":256208898},{"type":"v1","addr":"172.21.15.64:6829","nonce":256208898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":256208898},{"type":"v1","addr":"172.21.15.64:6833","nonce":256208898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":256208898},{"type":"v1","addr":"172.21.15.64:6831","nonce":256208898}]},"public_addr":"172.21.15.64:6827/256208898","cluster_addr":"172.21.15.64:6829/256208898","heartbeat_back_addr":"172.21.15.64:6833/256208898","heartbeat_front_addr":"172.21.15.64:6831/256208898","state":["exists","up"]},{"osd":4,"uuid":"1a5c7c64-fbb4-4d8b-b3d5-144164369304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":3371979738},{"type":"v1","addr":"172.21.15.176:6801","nonce":3371979738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":3371979738},{"type":"v1","addr":"172.21.15.176:6803","nonce":3371979738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":3371979738},{"type":"v1","addr":"172.21.15.176:6807","nonce":3371979738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":3371979738},{"type":"v1","addr":"172.21.15.176:6805","nonce":3371979738}]},"public_addr":"172.21.15.176:6801/3371979738","cluster_addr":"172.21.15.176:6803/3371979738","heartbeat_back_addr":"172.21.15.176:6807/3371979738","heartbeat_front_addr":"172.21.15.176:6805/3371979738","state":["exists","up"]},{"osd":5,"uuid":"dd30b1ea-5f0f-4f4a-8f3c-c1c35514acd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":2732967935},{"type":"v1","addr":"172.21.15.176:6809","nonce":2732967935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2732967935},{"type":"v1","addr":"172.21.15.176:6811","nonce":2732967935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2732967935},{"type":"v1","addr":"172.21.15.176:6815","nonce":2732967935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2732967935},{"type":"v1","addr":"172.21.15.176:6813","nonce":2732967935}]},"public_addr":"172.21.15.176:6809/2732967935","cluster_addr":"172.21.15.176:6811/2732967935","heartbeat_back_addr":"172.21.15.176:6815/2732967935","heartbeat_front_addr":"172.21.15.176:6813/2732967935","state":["exists","up"]},{"osd":6,"uuid":"093e0c53-f0e6-43c3-9d9d-141bc456784d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":937306186},{"type":"v1","addr":"172.21.15.176:6817","nonce":937306186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":937306186},{"type":"v1","addr":"172.21.15.176:6819","nonce":937306186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":937306186},{"type":"v1","addr":"172.21.15.176:6823","nonce":937306186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":937306186},{"type":"v1","addr":"172.21.15.176:6821","nonce":937306186}]},"public_addr":"172.21.15.176:6817/937306186","cluster_addr":"172.21.15.176:6819/937306186","heartbeat_back_addr":"172.21.15.176:6823/937306186","heartbeat_front_addr":"172.21.15.176:6821/937306186","state":["exists","up"]},{"osd":7,"uuid":"ef5d6516-1760-43db-bc37-a262e81466a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":3087207983},{"type":"v1","addr":"172.21.15.176:6825","nonce":3087207983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":3087207983},{"type":"v1","addr":"172.21.15.176:6827","nonce":3087207983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":3087207983},{"type":"v1","addr":"172.21.15.176:6831","nonce":3087207983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":3087207983},{"type":"v1","addr":"172.21.15.176:6829","nonce":3087207983}]},"public_addr":"172.21.15.176:6825/3087207983","cluster_addr":"172.21.15.176:6827/3087207983","heartbeat_back_addr":"172.21.15.176:6831/3087207983","heartbeat_front_addr":"172.21.15.176:6829/3087207983","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:51:44.259281+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:52:10.175114+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:52:35.921506+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:53:02.271692+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:53:26.046322+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:53:51.003242+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:54:16.120316+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:54:42.320346+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/3293519534":"2024-02-11T05:50:23.907719+0000","172.21.15.64:6801/4075411303":"2024-02-11T05:50:23.907719+0000","172.21.15.64:6800/4075411303":"2024-02-11T05:50:23.907719+0000","172.21.15.64:0/1368921613":"2024-02-11T05:49:56.679796+0000","172.21.15.64:0/3226601171":"2024-02-11T05:49:56.679796+0000","172.21.15.64:0/3433552474":"2024-02-11T05:49:56.679796+0000","172.21.15.64:6800/2493684466":"2024-02-11T05:49:56.679796+0000","172.21.15.64:6801/2493684466":"2024-02-11T05:49:56.679796+0000","172.21.15.64:0/1919796165":"2024-02-11T05:50:23.907719+0000","172.21.15.64:0/77145140":"2024-02-11T05:50:23.907719+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T05:55:32.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:32 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/4101459687' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:55:32.239 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:31 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:31.890+0000 7fe801586200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:55:32.239 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:32 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:32.022+0000 7fe801586200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:55:32.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:32 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/4101459687' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:55:32.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:32 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/4101459687' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:55:32.376 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:32 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:32.071+0000 7f680ee5c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:55:32.377 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:32 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:32.209+0000 7f680ee5c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:55:32.649 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:32 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:32.551+0000 7f680ee5c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:55:32.649 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:32 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:32.360+0000 7fe801586200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:55:32.649 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:32 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:32.451+0000 7fe801586200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:55:32.649 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:32 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:32.649+0000 7fe801586200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:55:32.707 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-10T05:55:32.708 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T05:55:32.954 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:55:32.988 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:32 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:32.836+0000 7fe801586200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:55:33.061 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:32 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:32.647+0000 7f680ee5c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:55:33.062 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:32 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:32.857+0000 7f680ee5c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:55:33.363 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:33 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:33.060+0000 7f680ee5c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:55:33.386 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:33 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:33.124+0000 7fe801586200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:55:33.386 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:33 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:33.385+0000 7fe801586200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:55:33.626 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:33 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:33.362+0000 7f680ee5c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:55:33.738 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:33 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:33.622+0000 7fe801586200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:55:34.053 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:33 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:33.625+0000 7f680ee5c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:55:34.053 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:33 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:33.872+0000 7f680ee5c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:55:34.738 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:34 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:34.425+0000 7fe801586200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:55:34.838 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:55:34.839 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":70,"fsid":"00e86dbe-c7d8-11ee-95b8-87774f69a715","created":"2024-02-10T05:49:26.002096+0000","modified":"2024-02-10T05:55:25.902709+0000","last_up_change":"2024-02-10T05:54:43.610359+0000","last_in_change":"2024-02-10T05:54:25.763533+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T05:52:38.253691+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-02-10T05:54:55.269909+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-02-10T05:54:55.517233+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-10T05:54:56.852965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-10T05:54:58.816169+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-10T05:55:00.886438+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"32df1361-a86f-44aa-bc33-d0b1e3fc140b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":3211593074},{"type":"v1","addr":"172.21.15.64:6803","nonce":3211593074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":3211593074},{"type":"v1","addr":"172.21.15.64:6805","nonce":3211593074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3211593074},{"type":"v1","addr":"172.21.15.64:6809","nonce":3211593074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":3211593074},{"type":"v1","addr":"172.21.15.64:6807","nonce":3211593074}]},"public_addr":"172.21.15.64:6803/3211593074","cluster_addr":"172.21.15.64:6805/3211593074","heartbeat_back_addr":"172.21.15.64:6809/3211593074","heartbeat_front_addr":"172.21.15.64:6807/3211593074","state":["exists","up"]},{"osd":1,"uuid":"cfe981c9-406d-4da9-89ff-81dbbef91183","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":614169574},{"type":"v1","addr":"172.21.15.64:6811","nonce":614169574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":614169574},{"type":"v1","addr":"172.21.15.64:6813","nonce":614169574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":614169574},{"type":"v1","addr":"172.21.15.64:6817","nonce":614169574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":614169574},{"type":"v1","addr":"172.21.15.64:6815","nonce":614169574}]},"public_addr":"172.21.15.64:6811/614169574","cluster_addr":"172.21.15.64:6813/614169574","heartbeat_back_addr":"172.21.15.64:6817/614169574","heartbeat_front_addr":"172.21.15.64:6815/614169574","state":["exists","up"]},{"osd":2,"uuid":"bb433070-a584-427e-bb43-bf20be79b123","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2362209772},{"type":"v1","addr":"172.21.15.64:6819","nonce":2362209772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2362209772},{"type":"v1","addr":"172.21.15.64:6821","nonce":2362209772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":2362209772},{"type":"v1","addr":"172.21.15.64:6825","nonce":2362209772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2362209772},{"type":"v1","addr":"172.21.15.64:6823","nonce":2362209772}]},"public_addr":"172.21.15.64:6819/2362209772","cluster_addr":"172.21.15.64:6821/2362209772","heartbeat_back_addr":"172.21.15.64:6825/2362209772","heartbeat_front_addr":"172.21.15.64:6823/2362209772","state":["exists","up"]},{"osd":3,"uuid":"bb31284d-6c1a-4f79-928a-bb758b45e196","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":256208898},{"type":"v1","addr":"172.21.15.64:6827","nonce":256208898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":256208898},{"type":"v1","addr":"172.21.15.64:6829","nonce":256208898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":256208898},{"type":"v1","addr":"172.21.15.64:6833","nonce":256208898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":256208898},{"type":"v1","addr":"172.21.15.64:6831","nonce":256208898}]},"public_addr":"172.21.15.64:6827/256208898","cluster_addr":"172.21.15.64:6829/256208898","heartbeat_back_addr":"172.21.15.64:6833/256208898","heartbeat_front_addr":"172.21.15.64:6831/256208898","state":["exists","up"]},{"osd":4,"uuid":"1a5c7c64-fbb4-4d8b-b3d5-144164369304","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":3371979738},{"type":"v1","addr":"172.21.15.176:6801","nonce":3371979738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":3371979738},{"type":"v1","addr":"172.21.15.176:6803","nonce":3371979738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":3371979738},{"type":"v1","addr":"172.21.15.176:6807","nonce":3371979738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":3371979738},{"type":"v1","addr":"172.21.15.176:6805","nonce":3371979738}]},"public_addr":"172.21.15.176:6801/3371979738","cluster_addr":"172.21.15.176:6803/3371979738","heartbeat_back_addr":"172.21.15.176:6807/3371979738","heartbeat_front_addr":"172.21.15.176:6805/3371979738","state":["exists","up"]},{"osd":5,"uuid":"dd30b1ea-5f0f-4f4a-8f3c-c1c35514acd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":2732967935},{"type":"v1","addr":"172.21.15.176:6809","nonce":2732967935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2732967935},{"type":"v1","addr":"172.21.15.176:6811","nonce":2732967935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2732967935},{"type":"v1","addr":"172.21.15.176:6815","nonce":2732967935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2732967935},{"type":"v1","addr":"172.21.15.176:6813","nonce":2732967935}]},"public_addr":"172.21.15.176:6809/2732967935","cluster_addr":"172.21.15.176:6811/2732967935","heartbeat_back_addr":"172.21.15.176:6815/2732967935","heartbeat_front_addr":"172.21.15.176:6813/2732967935","state":["exists","up"]},{"osd":6,"uuid":"093e0c53-f0e6-43c3-9d9d-141bc456784d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":937306186},{"type":"v1","addr":"172.21.15.176:6817","nonce":937306186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":937306186},{"type":"v1","addr":"172.21.15.176:6819","nonce":937306186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":937306186},{"type":"v1","addr":"172.21.15.176:6823","nonce":937306186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":937306186},{"type":"v1","addr":"172.21.15.176:6821","nonce":937306186}]},"public_addr":"172.21.15.176:6817/937306186","cluster_addr":"172.21.15.176:6819/937306186","heartbeat_back_addr":"172.21.15.176:6823/937306186","heartbeat_front_addr":"172.21.15.176:6821/937306186","state":["exists","up"]},{"osd":7,"uuid":"ef5d6516-1760-43db-bc37-a262e81466a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":3087207983},{"type":"v1","addr":"172.21.15.176:6825","nonce":3087207983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":3087207983},{"type":"v1","addr":"172.21.15.176:6827","nonce":3087207983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":3087207983},{"type":"v1","addr":"172.21.15.176:6831","nonce":3087207983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":3087207983},{"type":"v1","addr":"172.21.15.176:6829","nonce":3087207983}]},"public_addr":"172.21.15.176:6825/3087207983","cluster_addr":"172.21.15.176:6827/3087207983","heartbeat_back_addr":"172.21.15.176:6831/3087207983","heartbeat_front_addr":"172.21.15.176:6829/3087207983","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:51:44.259281+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:52:10.175114+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:52:35.921506+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:53:02.271692+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:53:26.046322+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:53:51.003242+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:54:16.120316+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:54:42.320346+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:0/3293519534":"2024-02-11T05:50:23.907719+0000","172.21.15.64:6801/4075411303":"2024-02-11T05:50:23.907719+0000","172.21.15.64:6800/4075411303":"2024-02-11T05:50:23.907719+0000","172.21.15.64:0/1368921613":"2024-02-11T05:49:56.679796+0000","172.21.15.64:0/3226601171":"2024-02-11T05:49:56.679796+0000","172.21.15.64:0/3433552474":"2024-02-11T05:49:56.679796+0000","172.21.15.64:6800/2493684466":"2024-02-11T05:49:56.679796+0000","172.21.15.64:6801/2493684466":"2024-02-11T05:49:56.679796+0000","172.21.15.64:0/1919796165":"2024-02-11T05:50:23.907719+0000","172.21.15.64:0/77145140":"2024-02-11T05:50:23.907719+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T05:55:35.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:34 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/2215249150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:55:35.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:34 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/2215249150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:55:35.126 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:34 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:34.727+0000 7f680ee5c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:55:35.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:34 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/2215249150' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:55:35.239 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:35.003+0000 7fe801586200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:55:35.239 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:35.101+0000 7fe801586200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:55:35.425 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:35.320+0000 7f680ee5c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:55:35.426 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:35.423+0000 7f680ee5c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:55:35.572 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-10T05:55:35.572 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-10T05:55:35.573 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-10T05:55:35.573 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-10T05:55:35.574 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-10T05:55:35.574 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-10T05:55:35.574 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-10T05:55:35.575 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-10T05:55:35.738 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:35.320+0000 7fe801586200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:55:35.738 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: [10/Feb/2024:05:55:35] ENGINE Bus STARTING 2024-02-10T05:55:35.738 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: CherryPy Checker: 2024-02-10T05:55:35.738 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: The Application mounted at '' has an empty config. 2024-02-10T05:55:35.739 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: 2024-02-10T05:55:35.739 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: [10/Feb/2024:05:55:35] ENGINE Serving on http://:::9283 2024-02-10T05:55:35.739 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x[32151]: [10/Feb/2024:05:55:35] ENGINE Bus STARTED 2024-02-10T05:55:35.996 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:35.706+0000 7f680ee5c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:55:35.996 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: [10/Feb/2024:05:55:35] ENGINE Bus STARTING 2024-02-10T05:55:35.997 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: Standby manager daemon x restarted 2024-02-10T05:55:35.997 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: Standby manager daemon x started 2024-02-10T05:55:35.997 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: from='mgr.? 172.21.15.176:0/2498824672' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T05:55:35.997 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: from='mgr.? 172.21.15.176:0/2498824672' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:55:35.997 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: from='mgr.? 172.21.15.176:0/2498824672' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T05:55:35.997 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: from='mgr.? 172.21.15.176:0/2498824672' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:55:35.997 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: Active manager daemon y restarted 2024-02-10T05:55:35.997 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: Activating manager daemon y 2024-02-10T05:55:35.997 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: osdmap e71: 8 total, 8 up, 8 in 2024-02-10T05:55:35.998 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: mgrmap e18: y(active, starting, since 0.0153181s), standbys: x 2024-02-10T05:55:35.998 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:55:35.998 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:55:35.998 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:55:35.998 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T05:55:35.998 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T05:55:35.998 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:55:35.998 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:55:35.998 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:55:35.998 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:55:35.998 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:55:35.998 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:55:35.998 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:55:35.998 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:55:35.999 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:55:35.999 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:55:35.999 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:55:35.999 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[38958]: Manager daemon y is now available 2024-02-10T05:55:35.999 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: Standby manager daemon x restarted 2024-02-10T05:55:35.999 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: Standby manager daemon x started 2024-02-10T05:55:35.999 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: from='mgr.? 172.21.15.176:0/2498824672' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T05:55:35.999 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: from='mgr.? 172.21.15.176:0/2498824672' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:55:35.999 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: from='mgr.? 172.21.15.176:0/2498824672' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T05:55:35.999 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: from='mgr.? 172.21.15.176:0/2498824672' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:55:35.999 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: Active manager daemon y restarted 2024-02-10T05:55:36.000 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: Activating manager daemon y 2024-02-10T05:55:36.000 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: osdmap e71: 8 total, 8 up, 8 in 2024-02-10T05:55:36.000 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: mgrmap e18: y(active, starting, since 0.0153181s), standbys: x 2024-02-10T05:55:36.000 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:55:36.000 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:55:36.000 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:55:36.000 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T05:55:36.000 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T05:55:36.000 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:55:36.000 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:55:36.000 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:55:36.000 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:55:36.000 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:55:36.001 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:55:36.001 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:55:36.001 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:55:36.001 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:55:36.001 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:55:36.001 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:55:36.001 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:35 smithi064 ceph-mon[25185]: Manager daemon y is now available 2024-02-10T05:55:36.139 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:55:36.181 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:55:36.185 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:55:36.185 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:55:36.185 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: Standby manager daemon x restarted 2024-02-10T05:55:36.185 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: Standby manager daemon x started 2024-02-10T05:55:36.185 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: from='mgr.? 172.21.15.176:0/2498824672' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T05:55:36.185 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: from='mgr.? 172.21.15.176:0/2498824672' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:55:36.186 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: from='mgr.? 172.21.15.176:0/2498824672' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T05:55:36.186 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: from='mgr.? 172.21.15.176:0/2498824672' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:55:36.186 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: Active manager daemon y restarted 2024-02-10T05:55:36.186 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: Activating manager daemon y 2024-02-10T05:55:36.186 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: osdmap e71: 8 total, 8 up, 8 in 2024-02-10T05:55:36.186 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: mgrmap e18: y(active, starting, since 0.0153181s), standbys: x 2024-02-10T05:55:36.186 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:55:36.186 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:55:36.186 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:55:36.186 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T05:55:36.186 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T05:55:36.186 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:55:36.186 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:55:36.186 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:55:36.187 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-10T05:55:36.187 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-10T05:55:36.187 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-10T05:55:36.187 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-10T05:55:36.187 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-10T05:55:36.187 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:55:36.187 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:55:36.187 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:55:36.187 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:35 smithi176 ceph-mon[30294]: Manager daemon y is now available 2024-02-10T05:55:36.190 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:55:36.190 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:55:36.193 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:55:36.196 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:55:36.255 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:36 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: CherryPy Checker: 2024-02-10T05:55:36.255 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:36 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: The Application mounted at '' has an empty config. 2024-02-10T05:55:36.255 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:36 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: 2024-02-10T05:55:36.255 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:36 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: [10/Feb/2024:05:55:36] ENGINE Serving on http://:::9283 2024-02-10T05:55:36.255 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:36 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: [10/Feb/2024:05:55:36] ENGINE Bus STARTED 2024-02-10T05:55:37.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:36 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:37.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:36 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:55:37.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:36 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:37.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:36 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:55:37.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:36 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:55:37.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:36 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:55:37.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:36 smithi064 ceph-mon[38958]: [10/Feb/2024:05:55:36] ENGINE Bus STARTING 2024-02-10T05:55:37.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:36 smithi064 ceph-mon[38958]: [10/Feb/2024:05:55:36] ENGINE Serving on http://172.21.15.64:8765 2024-02-10T05:55:37.127 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:36 smithi064 ceph-mon[38958]: mgrmap e19: y(active, since 1.01395s), standbys: x 2024-02-10T05:55:37.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:36 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:37.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:36 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:55:37.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:36 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:37.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:36 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:55:37.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:36 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:55:37.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:36 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:55:37.127 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:36 smithi064 ceph-mon[25185]: [10/Feb/2024:05:55:36] ENGINE Bus STARTING 2024-02-10T05:55:37.128 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:36 smithi064 ceph-mon[25185]: [10/Feb/2024:05:55:36] ENGINE Serving on http://172.21.15.64:8765 2024-02-10T05:55:37.128 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:36 smithi064 ceph-mon[25185]: mgrmap e19: y(active, since 1.01395s), standbys: x 2024-02-10T05:55:37.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:36 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:37.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:36 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:55:37.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:36 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:55:37.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:36 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:55:37.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:36 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:55:37.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:36 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:55:37.231 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:36 smithi176 ceph-mon[30294]: [10/Feb/2024:05:55:36] ENGINE Bus STARTING 2024-02-10T05:55:37.231 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:36 smithi176 ceph-mon[30294]: [10/Feb/2024:05:55:36] ENGINE Serving on http://172.21.15.64:8765 2024-02-10T05:55:37.231 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:36 smithi176 ceph-mon[30294]: mgrmap e19: y(active, since 1.01395s), standbys: x 2024-02-10T05:55:38.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:37 smithi176 ceph-mon[30294]: [10/Feb/2024:05:55:36] ENGINE Serving on https://172.21.15.64:7150 2024-02-10T05:55:38.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:37 smithi176 ceph-mon[30294]: [10/Feb/2024:05:55:36] ENGINE Bus STARTED 2024-02-10T05:55:38.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:37 smithi176 ceph-mon[30294]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:55:38.249 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:37 smithi064 ceph-mon[25185]: [10/Feb/2024:05:55:36] ENGINE Serving on https://172.21.15.64:7150 2024-02-10T05:55:38.249 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:37 smithi064 ceph-mon[25185]: [10/Feb/2024:05:55:36] ENGINE Bus STARTED 2024-02-10T05:55:38.249 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:37 smithi064 ceph-mon[25185]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:55:38.249 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:37 smithi064 ceph-mon[38958]: [10/Feb/2024:05:55:36] ENGINE Serving on https://172.21.15.64:7150 2024-02-10T05:55:38.249 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:37 smithi064 ceph-mon[38958]: [10/Feb/2024:05:55:36] ENGINE Bus STARTED 2024-02-10T05:55:38.250 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:37 smithi064 ceph-mon[38958]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:55:38.812 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:55:38 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[66249]: debug there is no tcmu-runner data available 2024-02-10T05:55:39.083 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:38 smithi064 ceph-mon[25185]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:55:39.083 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:38 smithi064 ceph-mon[25185]: mgrmap e20: y(active, since 2s), standbys: x 2024-02-10T05:55:39.083 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:38 smithi064 ceph-mon[25185]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:55:39.083 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:38 smithi064 ceph-mon[38958]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:55:39.083 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:38 smithi064 ceph-mon[38958]: mgrmap e20: y(active, since 2s), standbys: x 2024-02-10T05:55:39.083 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:38 smithi064 ceph-mon[38958]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:55:39.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:38 smithi176 ceph-mon[30294]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:55:39.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:38 smithi176 ceph-mon[30294]: mgrmap e20: y(active, since 2s), standbys: x 2024-02-10T05:55:39.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:38 smithi176 ceph-mon[30294]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:55:40.407 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:40 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:55:40.407 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:40 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:55:40.407 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:40 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:55:40.407 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:40 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:55:40.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:40 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:55:40.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:40 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:55:41.376 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:41 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: ::ffff:172.21.15.176 - - [10/Feb/2024:05:55:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T05:55:41.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:41 smithi064 ceph-mon[25185]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:55:41.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:41 smithi064 ceph-mon[25185]: mgrmap e21: y(active, since 4s), standbys: x 2024-02-10T05:55:41.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:41 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:55:41.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:41 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:55:41.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:41 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:41.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:41 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:41.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:41 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:41.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:41 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:41.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:41 smithi064 ceph-mon[25185]: Adjusting osd_memory_target on smithi176 to 3760M 2024-02-10T05:55:41.377 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:41 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:55:41.378 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:41 smithi064 ceph-mon[38958]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:55:41.378 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:41 smithi064 ceph-mon[38958]: mgrmap e21: y(active, since 4s), standbys: x 2024-02-10T05:55:41.378 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:41 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:55:41.378 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:41 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:55:41.378 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:41 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:41.378 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:41 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:41.378 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:41 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:41.378 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:41 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:41.378 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:41 smithi064 ceph-mon[38958]: Adjusting osd_memory_target on smithi176 to 3760M 2024-02-10T05:55:41.378 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:41 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:55:41.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:41 smithi176 ceph-mon[30294]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-02-10T05:55:41.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:41 smithi176 ceph-mon[30294]: mgrmap e21: y(active, since 4s), standbys: x 2024-02-10T05:55:41.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:41 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:55:41.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:41 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:55:41.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:41 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:41.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:41 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:41.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:41 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:41.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:41 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:55:41.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:41 smithi176 ceph-mon[30294]: Adjusting osd_memory_target on smithi176 to 3760M 2024-02-10T05:55:41.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:41 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:55:43.467 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:43 smithi064 ceph-mon[25185]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-02-10T05:55:43.468 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:43 smithi064 ceph-mon[38958]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-02-10T05:55:43.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:43 smithi176 ceph-mon[30294]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-02-10T05:55:45.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:45 smithi176 ceph-mon[30294]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-10T05:55:45.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:45 smithi064 ceph-mon[25185]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-10T05:55:45.591 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:45 smithi064 ceph-mon[38958]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-10T05:55:46.879 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:46 smithi064 ceph-mon[25185]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T05:55:46.879 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:46 smithi064 ceph-mon[38958]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T05:55:46.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:46 smithi176 ceph-mon[30294]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-02-10T05:55:48.479 INFO:teuthology.orchestra.run.smithi064.stdout:231928233998 2024-02-10T05:55:48.479 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-10T05:55:48.833 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:55:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[66249]: debug there is no tcmu-runner data available 2024-02-10T05:55:48.895 INFO:teuthology.orchestra.run.smithi064.stdout:55834574893 2024-02-10T05:55:48.895 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-10T05:55:49.120 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:48 smithi064 ceph-mon[38958]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-10T05:55:49.120 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:48 smithi064 ceph-mon[38958]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:55:49.120 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:48 smithi064 ceph-mon[25185]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-10T05:55:49.120 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:48 smithi064 ceph-mon[25185]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:55:49.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:48 smithi176 ceph-mon[30294]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-10T05:55:49.239 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:48 smithi176 ceph-mon[30294]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:55:49.627 INFO:teuthology.orchestra.run.smithi064.stdout:163208757272 2024-02-10T05:55:49.627 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-10T05:55:50.635 INFO:teuthology.orchestra.run.smithi064.stdout:77309411368 2024-02-10T05:55:50.636 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-10T05:55:51.066 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:50 smithi064 ceph-mon[25185]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T05:55:51.067 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:50 smithi064 ceph-mon[38958]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T05:55:51.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:50 smithi176 ceph-mon[30294]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T05:55:51.341 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:55:51 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: ::ffff:172.21.15.176 - - [10/Feb/2024:05:55:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T05:55:51.453 INFO:teuthology.orchestra.run.smithi064.stdout:128849018909 2024-02-10T05:55:51.453 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-10T05:55:52.073 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:51 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:55:52.074 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:51 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:55:52.101 INFO:teuthology.orchestra.run.smithi064.stdout:107374182435 2024-02-10T05:55:52.102 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-10T05:55:52.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:51 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:55:52.802 INFO:teuthology.orchestra.run.smithi064.stdout:34359738419 2024-02-10T05:55:52.803 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-10T05:55:52.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:52 smithi176 ceph-mon[30294]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T05:55:53.107 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:52 smithi064 ceph-mon[25185]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T05:55:53.107 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:52 smithi064 ceph-mon[38958]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T05:55:53.424 INFO:teuthology.orchestra.run.smithi064.stdout:197568495637 2024-02-10T05:55:53.424 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-10T05:55:53.475 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:55:53.504 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:55:54.725 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:55:54.726 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:55:54.726 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:55:54.726 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:55:54.727 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:55:54.856 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:54 smithi064 ceph-mon[25185]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:55:54.856 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:54 smithi064 ceph-mon[38958]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:55:55.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:54 smithi176 ceph-mon[30294]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:55:56.754 INFO:teuthology.orchestra.run.smithi064.stdout:55834574895 2024-02-10T05:55:56.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:56 smithi064 ceph-mon[38958]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:55:56.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:56 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/844883505' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T05:55:56.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:56 smithi064 ceph-mon[25185]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:55:56.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:56 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/844883505' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T05:55:56.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:56 smithi176 ceph-mon[30294]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:55:56.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:56 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/844883505' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T05:55:58.836 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:55:58 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[66249]: debug there is no tcmu-runner data available 2024-02-10T05:55:59.058 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:58 smithi064 ceph-mon[25185]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:55:59.058 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:55:58 smithi064 ceph-mon[25185]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:55:59.058 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:58 smithi064 ceph-mon[38958]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:55:59.058 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:55:58 smithi064 ceph-mon[38958]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:55:59.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:58 smithi176 ceph-mon[30294]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:55:59.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:55:58 smithi176 ceph-mon[30294]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:56:00.188 INFO:teuthology.orchestra.run.smithi064.stdout:163208757275 2024-02-10T05:56:00.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:00 smithi064 ceph-mon[25185]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:00.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:00 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/1725120881' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T05:56:00.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:00 smithi064 ceph-mon[38958]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:00.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:00 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/1725120881' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T05:56:01.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:00 smithi176 ceph-mon[30294]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:01.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:00 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/1725120881' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-10T05:56:01.626 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:01 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: ::ffff:172.21.15.176 - - [10/Feb/2024:05:56:01] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T05:56:03.025 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:02 smithi064 ceph-mon[25185]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:03.026 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:02 smithi064 ceph-mon[38958]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:03.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:02 smithi176 ceph-mon[30294]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:03.654 INFO:teuthology.orchestra.run.smithi064.stdout:107374182438 2024-02-10T05:56:03.885 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757272 got 163208757275 for osd.5 2024-02-10T05:56:03.885 DEBUG:teuthology.parallel:result is None 2024-02-10T05:56:04.056 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:56:04.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:03 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/126182126' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T05:56:04.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:03 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/126182126' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T05:56:04.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:03 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/126182126' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-10T05:56:04.288 INFO:teuthology.orchestra.run.smithi064.stdout:128849018913 2024-02-10T05:56:05.008 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:04 smithi064 ceph-mon[25185]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:05.008 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:04 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/3059133065' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T05:56:05.008 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:04 smithi064 ceph-mon[38958]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:05.008 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:04 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/3059133065' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T05:56:05.041 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182435 got 107374182438 for osd.3 2024-02-10T05:56:05.041 DEBUG:teuthology.parallel:result is None 2024-02-10T05:56:05.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:04 smithi176 ceph-mon[30294]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:05.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:04 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/3059133065' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-10T05:56:05.365 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574893 got 55834574895 for osd.1 2024-02-10T05:56:05.365 DEBUG:teuthology.parallel:result is None 2024-02-10T05:56:06.659 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:06 smithi064 ceph-mon[25185]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:06.660 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:06 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:56:06.660 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:06 smithi064 ceph-mon[38958]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:06.660 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:06 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:56:06.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:06 smithi176 ceph-mon[30294]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:06.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:06 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:56:07.310 INFO:teuthology.orchestra.run.smithi064.stdout:231928234002 2024-02-10T05:56:07.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:07 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/1805114503' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T05:56:07.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:07 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/1805114503' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T05:56:07.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:07 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/1805114503' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-10T05:56:08.245 INFO:teuthology.orchestra.run.smithi064.stdout:34359738422 2024-02-10T05:56:08.556 INFO:teuthology.orchestra.run.smithi064.stdout:197568495639 2024-02-10T05:56:08.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:08 smithi064 ceph-mon[25185]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:08.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:08 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/3928936057' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T05:56:08.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:08 smithi064 ceph-mon[25185]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:56:08.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:08 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/221225951' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T05:56:08.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:08 smithi064 ceph-mon[38958]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:08.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:08 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/3928936057' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T05:56:08.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:08 smithi064 ceph-mon[38958]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:56:08.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:08 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/221225951' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T05:56:08.988 INFO:teuthology.orchestra.run.smithi064.stdout:77309411372 2024-02-10T05:56:08.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:56:08 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[66249]: debug there is no tcmu-runner data available 2024-02-10T05:56:08.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:08 smithi176 ceph-mon[30294]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:08.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:08 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/3928936057' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T05:56:08.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:08 smithi176 ceph-mon[30294]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:56:08.989 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:08 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/221225951' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-10T05:56:09.379 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495637 got 197568495639 for osd.6 2024-02-10T05:56:09.380 DEBUG:teuthology.parallel:result is None 2024-02-10T05:56:09.823 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738419 got 34359738422 for osd.0 2024-02-10T05:56:09.823 DEBUG:teuthology.parallel:result is None 2024-02-10T05:56:09.909 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:09 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/3914192532' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T05:56:09.909 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:09 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/3914192532' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T05:56:09.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:09 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/3914192532' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T05:56:10.199 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018909 got 128849018913 for osd.4 2024-02-10T05:56:10.199 DEBUG:teuthology.parallel:result is None 2024-02-10T05:56:10.848 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233998 got 231928234002 for osd.7 2024-02-10T05:56:10.849 DEBUG:teuthology.parallel:result is None 2024-02-10T05:56:11.267 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411368 got 77309411372 for osd.2 2024-02-10T05:56:11.267 DEBUG:teuthology.parallel:result is None 2024-02-10T05:56:11.267 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-10T05:56:11.268 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T05:56:11.493 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:11 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: ::ffff:172.21.15.176 - - [10/Feb/2024:05:56:11] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T05:56:11.514 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:56:11.769 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:11 smithi064 ceph-mon[25185]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:11.769 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:11 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:11.769 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:11 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:11.770 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:11 smithi064 ceph-mon[38958]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:11.770 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:11 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:11.770 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:11 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:11.790 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:11 smithi176 ceph-mon[30294]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:11.790 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:11 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:11.790 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:11 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:13.502 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:56:13.505 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-02-10T05:56:13.626 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[38958]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:13.626 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:13.626 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:13.626 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:13.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:13.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:13.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:13.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[38958]: Adjusting osd_memory_target on smithi064 to 3760M 2024-02-10T05:56:13.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:13.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:13.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:13.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[38958]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-10T05:56:13.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[38958]: Updating smithi176:/etc/ceph/ceph.conf 2024-02-10T05:56:13.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[38958]: Updating smithi064:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:56:13.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:13.627 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:13.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:13.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:13.628 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:13.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[25185]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:13.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:13.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:13.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:13.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:13.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:13.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:13.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[25185]: Adjusting osd_memory_target on smithi064 to 3760M 2024-02-10T05:56:13.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:13.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:13.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:13.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[25185]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-10T05:56:13.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[25185]: Updating smithi176:/etc/ceph/ceph.conf 2024-02-10T05:56:13.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[25185]: Updating smithi064:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:56:13.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:13.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:13.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:13.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:13.629 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:13 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:13.738 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:13 smithi176 ceph-mon[30294]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:13.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:13 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:13.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:13 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:13.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:13 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:13.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:13 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:13.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:13 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:13.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:13 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:56:13.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:13 smithi176 ceph-mon[30294]: Adjusting osd_memory_target on smithi064 to 3760M 2024-02-10T05:56:13.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:13 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:13.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:13 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:56:13.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:13 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:56:13.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:13 smithi176 ceph-mon[30294]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-10T05:56:13.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:13 smithi176 ceph-mon[30294]: Updating smithi176:/etc/ceph/ceph.conf 2024-02-10T05:56:13.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:13 smithi176 ceph-mon[30294]: Updating smithi064:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:56:13.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:13 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:13.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:13 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:13.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:13 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:13.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:13 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:13.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:13 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:14.220 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-02-10T05:56:11.785722+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1320,"num_read_kb":1131,"num_write":873,"num_write_kb":4027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1771654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":854,"ondisk_log_size":854,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749928448,"kb_used":240784,"kb_used_data":7996,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749687664,"statfs":{"total":767926730752,"available":767680167936,"internally_reserved":0,"allocated":8187904,"data_stored":4433876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003403"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735275+0000","last_change":"2024-02-10T05:55:02.583157+0000","last_active":"2024-02-10T05:55:35.735275+0000","last_peered":"2024-02-10T05:55:35.735275+0000","last_clean":"2024-02-10T05:55:35.735275+0000","last_became_active":"2024-02-10T05:55:02.582860+0000","last_became_peered":"2024-02-10T05:55:02.582860+0000","last_unstale":"2024-02-10T05:55:35.735275+0000","last_undegraded":"2024-02-10T05:55:35.735275+0000","last_fullsized":"2024-02-10T05:55:35.735275+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:40:47.915444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.745577+0000","last_change":"2024-02-10T05:55:00.573270+0000","last_active":"2024-02-10T05:55:35.745577+0000","last_peered":"2024-02-10T05:55:35.745577+0000","last_clean":"2024-02-10T05:55:35.745577+0000","last_became_active":"2024-02-10T05:55:00.572941+0000","last_became_peered":"2024-02-10T05:55:00.572941+0000","last_unstale":"2024-02-10T05:55:35.745577+0000","last_undegraded":"2024-02-10T05:55:35.745577+0000","last_fullsized":"2024-02-10T05:55:35.745577+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:47:13.730741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.740323+0000","last_change":"2024-02-10T05:54:56.533959+0000","last_active":"2024-02-10T05:55:35.740323+0000","last_peered":"2024-02-10T05:55:35.740323+0000","last_clean":"2024-02-10T05:55:35.740323+0000","last_became_active":"2024-02-10T05:54:56.533570+0000","last_became_peered":"2024-02-10T05:54:56.533570+0000","last_unstale":"2024-02-10T05:55:35.740323+0000","last_undegraded":"2024-02-10T05:55:35.740323+0000","last_fullsized":"2024-02-10T05:55:35.740323+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:56:22.136122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'15","reported_seq":75,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735029+0000","last_change":"2024-02-10T05:55:25.937510+0000","last_active":"2024-02-10T05:55:35.735029+0000","last_peered":"2024-02-10T05:55:35.735029+0000","last_clean":"2024-02-10T05:55:35.735029+0000","last_became_active":"2024-02-10T05:55:25.924844+0000","last_became_peered":"2024-02-10T05:55:25.924844+0000","last_unstale":"2024-02-10T05:55:35.735029+0000","last_undegraded":"2024-02-10T05:55:35.735029+0000","last_fullsized":"2024-02-10T05:55:35.735029+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:44:00.040076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338286+0000","last_change":"2024-02-10T05:55:03.129505+0000","last_active":"2024-02-10T05:55:36.338286+0000","last_peered":"2024-02-10T05:55:36.338286+0000","last_clean":"2024-02-10T05:55:36.338286+0000","last_became_active":"2024-02-10T05:55:03.129113+0000","last_became_peered":"2024-02-10T05:55:03.129113+0000","last_unstale":"2024-02-10T05:55:36.338286+0000","last_undegraded":"2024-02-10T05:55:36.338286+0000","last_fullsized":"2024-02-10T05:55:36.338286+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:54:52.858555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735034+0000","last_change":"2024-02-10T05:55:00.573765+0000","last_active":"2024-02-10T05:55:35.735034+0000","last_peered":"2024-02-10T05:55:35.735034+0000","last_clean":"2024-02-10T05:55:35.735034+0000","last_became_active":"2024-02-10T05:55:00.573432+0000","last_became_peered":"2024-02-10T05:55:00.573432+0000","last_unstale":"2024-02-10T05:55:35.735034+0000","last_undegraded":"2024-02-10T05:55:35.735034+0000","last_fullsized":"2024-02-10T05:55:35.735034+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:39:15.660426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.339037+0000","last_change":"2024-02-10T05:54:56.543724+0000","last_active":"2024-02-10T05:55:36.339037+0000","last_peered":"2024-02-10T05:55:36.339037+0000","last_clean":"2024-02-10T05:55:36.339037+0000","last_became_active":"2024-02-10T05:54:56.543445+0000","last_became_peered":"2024-02-10T05:54:56.543445+0000","last_unstale":"2024-02-10T05:55:36.339037+0000","last_undegraded":"2024-02-10T05:55:36.339037+0000","last_fullsized":"2024-02-10T05:55:36.339037+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:07:42.871971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'39","reported_seq":102,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338199+0000","last_change":"2024-02-10T05:54:58.565969+0000","last_active":"2024-02-10T05:55:36.338199+0000","last_peered":"2024-02-10T05:55:36.338199+0000","last_clean":"2024-02-10T05:55:36.338199+0000","last_became_active":"2024-02-10T05:54:58.565454+0000","last_became_peered":"2024-02-10T05:54:58.565454+0000","last_unstale":"2024-02-10T05:55:36.338199+0000","last_undegraded":"2024-02-10T05:55:36.338199+0000","last_fullsized":"2024-02-10T05:55:36.338199+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:08:42.374580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.740792+0000","last_change":"2024-02-10T05:55:02.588648+0000","last_active":"2024-02-10T05:55:35.740792+0000","last_peered":"2024-02-10T05:55:35.740792+0000","last_clean":"2024-02-10T05:55:35.740792+0000","last_became_active":"2024-02-10T05:55:02.588008+0000","last_became_peered":"2024-02-10T05:55:02.588008+0000","last_unstale":"2024-02-10T05:55:35.740792+0000","last_undegraded":"2024-02-10T05:55:35.740792+0000","last_fullsized":"2024-02-10T05:55:35.740792+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:27:23.404755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014343+0000","last_change":"2024-02-10T05:55:00.571339+0000","last_active":"2024-02-10T05:55:36.014343+0000","last_peered":"2024-02-10T05:55:36.014343+0000","last_clean":"2024-02-10T05:55:36.014343+0000","last_became_active":"2024-02-10T05:55:00.571062+0000","last_became_peered":"2024-02-10T05:55:00.571062+0000","last_unstale":"2024-02-10T05:55:36.014343+0000","last_undegraded":"2024-02-10T05:55:36.014343+0000","last_fullsized":"2024-02-10T05:55:36.014343+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:44:05.542585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014613+0000","last_change":"2024-02-10T05:54:56.537716+0000","last_active":"2024-02-10T05:55:36.014613+0000","last_peered":"2024-02-10T05:55:36.014613+0000","last_clean":"2024-02-10T05:55:36.014613+0000","last_became_active":"2024-02-10T05:54:56.537413+0000","last_became_peered":"2024-02-10T05:54:56.537413+0000","last_unstale":"2024-02-10T05:55:36.014613+0000","last_undegraded":"2024-02-10T05:55:36.014613+0000","last_fullsized":"2024-02-10T05:55:36.014613+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:24:13.401436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'21","reported_seq":69,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735478+0000","last_change":"2024-02-10T05:54:58.565422+0000","last_active":"2024-02-10T05:55:35.735478+0000","last_peered":"2024-02-10T05:55:35.735478+0000","last_clean":"2024-02-10T05:55:35.735478+0000","last_became_active":"2024-02-10T05:54:58.564397+0000","last_became_peered":"2024-02-10T05:54:58.564397+0000","last_unstale":"2024-02-10T05:55:35.735478+0000","last_undegraded":"2024-02-10T05:55:35.735478+0000","last_fullsized":"2024-02-10T05:55:35.735478+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:28:10.134384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014213+0000","last_change":"2024-02-10T05:55:02.584058+0000","last_active":"2024-02-10T05:55:36.014213+0000","last_peered":"2024-02-10T05:55:36.014213+0000","last_clean":"2024-02-10T05:55:36.014213+0000","last_became_active":"2024-02-10T05:55:02.583691+0000","last_became_peered":"2024-02-10T05:55:02.583691+0000","last_unstale":"2024-02-10T05:55:36.014213+0000","last_undegraded":"2024-02-10T05:55:36.014213+0000","last_fullsized":"2024-02-10T05:55:36.014213+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:58:13.800252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.030938+0000","last_change":"2024-02-10T05:55:00.570365+0000","last_active":"2024-02-10T05:55:36.030938+0000","last_peered":"2024-02-10T05:55:36.030938+0000","last_clean":"2024-02-10T05:55:36.030938+0000","last_became_active":"2024-02-10T05:55:00.569976+0000","last_became_peered":"2024-02-10T05:55:00.569976+0000","last_unstale":"2024-02-10T05:55:36.030938+0000","last_undegraded":"2024-02-10T05:55:36.030938+0000","last_fullsized":"2024-02-10T05:55:36.030938+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:39:40.031844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014556+0000","last_change":"2024-02-10T05:54:56.539557+0000","last_active":"2024-02-10T05:55:36.014556+0000","last_peered":"2024-02-10T05:55:36.014556+0000","last_clean":"2024-02-10T05:55:36.014556+0000","last_became_active":"2024-02-10T05:54:56.539244+0000","last_became_peered":"2024-02-10T05:54:56.539244+0000","last_unstale":"2024-02-10T05:55:36.014556+0000","last_undegraded":"2024-02-10T05:55:36.014556+0000","last_fullsized":"2024-02-10T05:55:36.014556+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:06:03.603419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'11","reported_seq":51,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735320+0000","last_change":"2024-02-10T05:54:58.566132+0000","last_active":"2024-02-10T05:55:35.735320+0000","last_peered":"2024-02-10T05:55:35.735320+0000","last_clean":"2024-02-10T05:55:35.735320+0000","last_became_active":"2024-02-10T05:54:58.565645+0000","last_became_peered":"2024-02-10T05:54:58.565645+0000","last_unstale":"2024-02-10T05:55:35.735320+0000","last_undegraded":"2024-02-10T05:55:35.735320+0000","last_fullsized":"2024-02-10T05:55:35.735320+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:40:12.776793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735757+0000","last_change":"2024-02-10T05:55:03.131517+0000","last_active":"2024-02-10T05:55:35.735757+0000","last_peered":"2024-02-10T05:55:35.735757+0000","last_clean":"2024-02-10T05:55:35.735757+0000","last_became_active":"2024-02-10T05:55:03.129312+0000","last_became_peered":"2024-02-10T05:55:03.129312+0000","last_unstale":"2024-02-10T05:55:35.735757+0000","last_undegraded":"2024-02-10T05:55:35.735757+0000","last_fullsized":"2024-02-10T05:55:35.735757+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:32:26.382992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.745162+0000","last_change":"2024-02-10T05:55:00.568070+0000","last_active":"2024-02-10T05:55:35.745162+0000","last_peered":"2024-02-10T05:55:35.745162+0000","last_clean":"2024-02-10T05:55:35.745162+0000","last_became_active":"2024-02-10T05:55:00.567688+0000","last_became_peered":"2024-02-10T05:55:00.567688+0000","last_unstale":"2024-02-10T05:55:35.745162+0000","last_undegraded":"2024-02-10T05:55:35.745162+0000","last_fullsized":"2024-02-10T05:55:35.745162+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:11:25.503197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.744260+0000","last_change":"2024-02-10T05:54:56.537339+0000","last_active":"2024-02-10T05:55:35.744260+0000","last_peered":"2024-02-10T05:55:35.744260+0000","last_clean":"2024-02-10T05:55:35.744260+0000","last_became_active":"2024-02-10T05:54:56.537037+0000","last_became_peered":"2024-02-10T05:54:56.537037+0000","last_unstale":"2024-02-10T05:55:35.744260+0000","last_undegraded":"2024-02-10T05:55:35.744260+0000","last_fullsized":"2024-02-10T05:55:35.744260+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:19:54.965360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.027415+0000","last_change":"2024-02-10T05:54:58.560807+0000","last_active":"2024-02-10T05:55:36.027415+0000","last_peered":"2024-02-10T05:55:36.027415+0000","last_clean":"2024-02-10T05:55:36.027415+0000","last_became_active":"2024-02-10T05:54:58.560477+0000","last_became_peered":"2024-02-10T05:54:58.560477+0000","last_unstale":"2024-02-10T05:55:36.027415+0000","last_undegraded":"2024-02-10T05:55:36.027415+0000","last_fullsized":"2024-02-10T05:55:36.027415+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:26:46.496770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.339115+0000","last_change":"2024-02-10T05:55:03.129750+0000","last_active":"2024-02-10T05:55:36.339115+0000","last_peered":"2024-02-10T05:55:36.339115+0000","last_clean":"2024-02-10T05:55:36.339115+0000","last_became_active":"2024-02-10T05:55:03.129430+0000","last_became_peered":"2024-02-10T05:55:03.129430+0000","last_unstale":"2024-02-10T05:55:36.339115+0000","last_undegraded":"2024-02-10T05:55:36.339115+0000","last_fullsized":"2024-02-10T05:55:36.339115+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:22:50.301437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735954+0000","last_change":"2024-02-10T05:55:00.575265+0000","last_active":"2024-02-10T05:55:35.735954+0000","last_peered":"2024-02-10T05:55:35.735954+0000","last_clean":"2024-02-10T05:55:35.735954+0000","last_became_active":"2024-02-10T05:55:00.574722+0000","last_became_peered":"2024-02-10T05:55:00.574722+0000","last_unstale":"2024-02-10T05:55:35.735954+0000","last_undegraded":"2024-02-10T05:55:35.735954+0000","last_fullsized":"2024-02-10T05:55:35.735954+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:36:44.376095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735605+0000","last_change":"2024-02-10T05:54:56.541416+0000","last_active":"2024-02-10T05:55:35.735605+0000","last_peered":"2024-02-10T05:55:35.735605+0000","last_clean":"2024-02-10T05:55:35.735605+0000","last_became_active":"2024-02-10T05:54:56.541127+0000","last_became_peered":"2024-02-10T05:54:56.541127+0000","last_unstale":"2024-02-10T05:55:35.735605+0000","last_undegraded":"2024-02-10T05:55:35.735605+0000","last_fullsized":"2024-02-10T05:55:35.735605+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:06:08.251992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'18","reported_seq":64,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338797+0000","last_change":"2024-02-10T05:54:58.566311+0000","last_active":"2024-02-10T05:55:36.338797+0000","last_peered":"2024-02-10T05:55:36.338797+0000","last_clean":"2024-02-10T05:55:36.338797+0000","last_became_active":"2024-02-10T05:54:58.566016+0000","last_became_peered":"2024-02-10T05:54:58.566016+0000","last_unstale":"2024-02-10T05:55:36.338797+0000","last_undegraded":"2024-02-10T05:55:36.338797+0000","last_fullsized":"2024-02-10T05:55:36.338797+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:34:43.295518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.031259+0000","last_change":"2024-02-10T05:55:02.580987+0000","last_active":"2024-02-10T05:55:36.031259+0000","last_peered":"2024-02-10T05:55:36.031259+0000","last_clean":"2024-02-10T05:55:36.031259+0000","last_became_active":"2024-02-10T05:55:02.580621+0000","last_became_peered":"2024-02-10T05:55:02.580621+0000","last_unstale":"2024-02-10T05:55:36.031259+0000","last_undegraded":"2024-02-10T05:55:36.031259+0000","last_fullsized":"2024-02-10T05:55:36.031259+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:07:00.603547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"67'8","reported_seq":52,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:56:06.597577+0000","last_change":"2024-02-10T05:55:00.573585+0000","last_active":"2024-02-10T05:56:06.597577+0000","last_peered":"2024-02-10T05:56:06.597577+0000","last_clean":"2024-02-10T05:56:06.597577+0000","last_became_active":"2024-02-10T05:55:00.573247+0000","last_became_peered":"2024-02-10T05:55:00.573247+0000","last_unstale":"2024-02-10T05:56:06.597577+0000","last_undegraded":"2024-02-10T05:56:06.597577+0000","last_fullsized":"2024-02-10T05:56:06.597577+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:39:52.075233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.745659+0000","last_change":"2024-02-10T05:54:56.545093+0000","last_active":"2024-02-10T05:55:35.745659+0000","last_peered":"2024-02-10T05:55:35.745659+0000","last_clean":"2024-02-10T05:55:35.745659+0000","last_became_active":"2024-02-10T05:54:56.544798+0000","last_became_peered":"2024-02-10T05:54:56.544798+0000","last_unstale":"2024-02-10T05:55:35.745659+0000","last_undegraded":"2024-02-10T05:55:35.745659+0000","last_fullsized":"2024-02-10T05:55:35.745659+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:53:01.776249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'16","reported_seq":56,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.741760+0000","last_change":"2024-02-10T05:54:58.556890+0000","last_active":"2024-02-10T05:55:35.741760+0000","last_peered":"2024-02-10T05:55:35.741760+0000","last_clean":"2024-02-10T05:55:35.741760+0000","last_became_active":"2024-02-10T05:54:58.555915+0000","last_became_peered":"2024-02-10T05:54:58.555915+0000","last_unstale":"2024-02-10T05:55:35.741760+0000","last_undegraded":"2024-02-10T05:55:35.741760+0000","last_fullsized":"2024-02-10T05:55:35.741760+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:46:57.139563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.745323+0000","last_change":"2024-02-10T05:55:02.586369+0000","last_active":"2024-02-10T05:55:35.745323+0000","last_peered":"2024-02-10T05:55:35.745323+0000","last_clean":"2024-02-10T05:55:35.745323+0000","last_became_active":"2024-02-10T05:55:02.585359+0000","last_became_peered":"2024-02-10T05:55:02.585359+0000","last_unstale":"2024-02-10T05:55:35.745323+0000","last_undegraded":"2024-02-10T05:55:35.745323+0000","last_fullsized":"2024-02-10T05:55:35.745323+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:55:53.925628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.742163+0000","last_change":"2024-02-10T05:55:00.567096+0000","last_active":"2024-02-10T05:55:35.742163+0000","last_peered":"2024-02-10T05:55:35.742163+0000","last_clean":"2024-02-10T05:55:35.742163+0000","last_became_active":"2024-02-10T05:55:00.566767+0000","last_became_peered":"2024-02-10T05:55:00.566767+0000","last_unstale":"2024-02-10T05:55:35.742163+0000","last_undegraded":"2024-02-10T05:55:35.742163+0000","last_fullsized":"2024-02-10T05:55:35.742163+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:29:13.795369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.337945+0000","last_change":"2024-02-10T05:54:56.544008+0000","last_active":"2024-02-10T05:55:36.337945+0000","last_peered":"2024-02-10T05:55:36.337945+0000","last_clean":"2024-02-10T05:55:36.337945+0000","last_became_active":"2024-02-10T05:54:56.543695+0000","last_became_peered":"2024-02-10T05:54:56.543695+0000","last_unstale":"2024-02-10T05:55:36.337945+0000","last_undegraded":"2024-02-10T05:55:36.337945+0000","last_fullsized":"2024-02-10T05:55:36.337945+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:27:24.021139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'17","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338576+0000","last_change":"2024-02-10T05:54:58.559748+0000","last_active":"2024-02-10T05:55:36.338576+0000","last_peered":"2024-02-10T05:55:36.338576+0000","last_clean":"2024-02-10T05:55:36.338576+0000","last_became_active":"2024-02-10T05:54:58.559149+0000","last_became_peered":"2024-02-10T05:54:58.559149+0000","last_unstale":"2024-02-10T05:55:36.338576+0000","last_undegraded":"2024-02-10T05:55:36.338576+0000","last_fullsized":"2024-02-10T05:55:36.338576+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:55:47.488264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"67'1","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.030936+0000","last_change":"2024-02-10T05:55:02.581397+0000","last_active":"2024-02-10T05:55:36.030936+0000","last_peered":"2024-02-10T05:55:36.030936+0000","last_clean":"2024-02-10T05:55:36.030936+0000","last_became_active":"2024-02-10T05:55:02.580933+0000","last_became_peered":"2024-02-10T05:55:02.580933+0000","last_unstale":"2024-02-10T05:55:36.030936+0000","last_undegraded":"2024-02-10T05:55:36.030936+0000","last_fullsized":"2024-02-10T05:55:36.030936+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:43:14.953573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338025+0000","last_change":"2024-02-10T05:55:00.575266+0000","last_active":"2024-02-10T05:55:36.338025+0000","last_peered":"2024-02-10T05:55:36.338025+0000","last_clean":"2024-02-10T05:55:36.338025+0000","last_became_active":"2024-02-10T05:55:00.574983+0000","last_became_peered":"2024-02-10T05:55:00.574983+0000","last_unstale":"2024-02-10T05:55:36.338025+0000","last_undegraded":"2024-02-10T05:55:36.338025+0000","last_fullsized":"2024-02-10T05:55:36.338025+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:02:01.862287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.740406+0000","last_change":"2024-02-10T05:54:56.545934+0000","last_active":"2024-02-10T05:55:35.740406+0000","last_peered":"2024-02-10T05:55:35.740406+0000","last_clean":"2024-02-10T05:55:35.740406+0000","last_became_active":"2024-02-10T05:54:56.545572+0000","last_became_peered":"2024-02-10T05:54:56.545572+0000","last_unstale":"2024-02-10T05:55:35.740406+0000","last_undegraded":"2024-02-10T05:55:35.740406+0000","last_fullsized":"2024-02-10T05:55:35.740406+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:00:41.770228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":35,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.337992+0000","last_change":"2024-02-10T05:54:58.562130+0000","last_active":"2024-02-10T05:55:36.337992+0000","last_peered":"2024-02-10T05:55:36.337992+0000","last_clean":"2024-02-10T05:55:36.337992+0000","last_became_active":"2024-02-10T05:54:58.561805+0000","last_became_peered":"2024-02-10T05:54:58.561805+0000","last_unstale":"2024-02-10T05:55:36.337992+0000","last_undegraded":"2024-02-10T05:55:36.337992+0000","last_fullsized":"2024-02-10T05:55:36.337992+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:58:07.278752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338146+0000","last_change":"2024-02-10T05:55:03.129793+0000","last_active":"2024-02-10T05:55:36.338146+0000","last_peered":"2024-02-10T05:55:36.338146+0000","last_clean":"2024-02-10T05:55:36.338146+0000","last_became_active":"2024-02-10T05:55:03.129134+0000","last_became_peered":"2024-02-10T05:55:03.129134+0000","last_unstale":"2024-02-10T05:55:36.338146+0000","last_undegraded":"2024-02-10T05:55:36.338146+0000","last_fullsized":"2024-02-10T05:55:36.338146+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:11:59.629570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.031137+0000","last_change":"2024-02-10T05:55:00.573837+0000","last_active":"2024-02-10T05:55:36.031137+0000","last_peered":"2024-02-10T05:55:36.031137+0000","last_clean":"2024-02-10T05:55:36.031137+0000","last_became_active":"2024-02-10T05:55:00.573556+0000","last_became_peered":"2024-02-10T05:55:00.573556+0000","last_unstale":"2024-02-10T05:55:36.031137+0000","last_undegraded":"2024-02-10T05:55:36.031137+0000","last_fullsized":"2024-02-10T05:55:36.031137+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:01:52.648894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014806+0000","last_change":"2024-02-10T05:54:56.540211+0000","last_active":"2024-02-10T05:55:36.014806+0000","last_peered":"2024-02-10T05:55:36.014806+0000","last_clean":"2024-02-10T05:55:36.014806+0000","last_became_active":"2024-02-10T05:54:56.539924+0000","last_became_peered":"2024-02-10T05:54:56.539924+0000","last_unstale":"2024-02-10T05:55:36.014806+0000","last_undegraded":"2024-02-10T05:55:36.014806+0000","last_fullsized":"2024-02-10T05:55:36.014806+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:20:49.267434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"67'17","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338179+0000","last_change":"2024-02-10T05:54:58.562915+0000","last_active":"2024-02-10T05:55:36.338179+0000","last_peered":"2024-02-10T05:55:36.338179+0000","last_clean":"2024-02-10T05:55:36.338179+0000","last_became_active":"2024-02-10T05:54:58.562614+0000","last_became_peered":"2024-02-10T05:54:58.562614+0000","last_unstale":"2024-02-10T05:55:36.338179+0000","last_undegraded":"2024-02-10T05:55:36.338179+0000","last_fullsized":"2024-02-10T05:55:36.338179+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:14:19.106756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.738481+0000","last_change":"2024-02-10T05:55:02.588905+0000","last_active":"2024-02-10T05:55:35.738481+0000","last_peered":"2024-02-10T05:55:35.738481+0000","last_clean":"2024-02-10T05:55:35.738481+0000","last_became_active":"2024-02-10T05:55:02.588444+0000","last_became_peered":"2024-02-10T05:55:02.588444+0000","last_unstale":"2024-02-10T05:55:35.738481+0000","last_undegraded":"2024-02-10T05:55:35.738481+0000","last_fullsized":"2024-02-10T05:55:35.738481+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:47:18.063995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338402+0000","last_change":"2024-02-10T05:55:00.579149+0000","last_active":"2024-02-10T05:55:36.338402+0000","last_peered":"2024-02-10T05:55:36.338402+0000","last_clean":"2024-02-10T05:55:36.338402+0000","last_became_active":"2024-02-10T05:55:00.578524+0000","last_became_peered":"2024-02-10T05:55:00.578524+0000","last_unstale":"2024-02-10T05:55:36.338402+0000","last_undegraded":"2024-02-10T05:55:36.338402+0000","last_fullsized":"2024-02-10T05:55:36.338402+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:57:12.009589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.031335+0000","last_change":"2024-02-10T05:54:56.544187+0000","last_active":"2024-02-10T05:55:36.031335+0000","last_peered":"2024-02-10T05:55:36.031335+0000","last_clean":"2024-02-10T05:55:36.031335+0000","last_became_active":"2024-02-10T05:54:56.543884+0000","last_became_peered":"2024-02-10T05:54:56.543884+0000","last_unstale":"2024-02-10T05:55:36.031335+0000","last_undegraded":"2024-02-10T05:55:36.031335+0000","last_fullsized":"2024-02-10T05:55:36.031335+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:39:54.458211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.743270+0000","last_change":"2024-02-10T05:54:58.555538+0000","last_active":"2024-02-10T05:55:35.743270+0000","last_peered":"2024-02-10T05:55:35.743270+0000","last_clean":"2024-02-10T05:55:35.743270+0000","last_became_active":"2024-02-10T05:54:58.555218+0000","last_became_peered":"2024-02-10T05:54:58.555218+0000","last_unstale":"2024-02-10T05:55:35.743270+0000","last_undegraded":"2024-02-10T05:55:35.743270+0000","last_fullsized":"2024-02-10T05:55:35.743270+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:58:13.440955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.337967+0000","last_change":"2024-02-10T05:55:02.587907+0000","last_active":"2024-02-10T05:55:36.337967+0000","last_peered":"2024-02-10T05:55:36.337967+0000","last_clean":"2024-02-10T05:55:36.337967+0000","last_became_active":"2024-02-10T05:55:02.587611+0000","last_became_peered":"2024-02-10T05:55:02.587611+0000","last_unstale":"2024-02-10T05:55:36.337967+0000","last_undegraded":"2024-02-10T05:55:36.337967+0000","last_fullsized":"2024-02-10T05:55:36.337967+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:31:30.435093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.743056+0000","last_change":"2024-02-10T05:55:00.577369+0000","last_active":"2024-02-10T05:55:35.743056+0000","last_peered":"2024-02-10T05:55:35.743056+0000","last_clean":"2024-02-10T05:55:35.743056+0000","last_became_active":"2024-02-10T05:55:00.577073+0000","last_became_peered":"2024-02-10T05:55:00.577073+0000","last_unstale":"2024-02-10T05:55:35.743056+0000","last_undegraded":"2024-02-10T05:55:35.743056+0000","last_fullsized":"2024-02-10T05:55:35.743056+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:00:47.714540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.736227+0000","last_change":"2024-02-10T05:54:56.535861+0000","last_active":"2024-02-10T05:55:35.736227+0000","last_peered":"2024-02-10T05:55:35.736227+0000","last_clean":"2024-02-10T05:55:35.736227+0000","last_became_active":"2024-02-10T05:54:56.535593+0000","last_became_peered":"2024-02-10T05:54:56.535593+0000","last_unstale":"2024-02-10T05:55:35.736227+0000","last_undegraded":"2024-02-10T05:55:35.736227+0000","last_fullsized":"2024-02-10T05:55:35.736227+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:39:33.156246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735533+0000","last_change":"2024-02-10T05:54:58.563664+0000","last_active":"2024-02-10T05:55:35.735533+0000","last_peered":"2024-02-10T05:55:35.735533+0000","last_clean":"2024-02-10T05:55:35.735533+0000","last_became_active":"2024-02-10T05:54:58.562365+0000","last_became_peered":"2024-02-10T05:54:58.562365+0000","last_unstale":"2024-02-10T05:55:35.735533+0000","last_undegraded":"2024-02-10T05:55:35.735533+0000","last_fullsized":"2024-02-10T05:55:35.735533+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:17:29.418689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.741042+0000","last_change":"2024-02-10T05:55:02.585145+0000","last_active":"2024-02-10T05:55:35.741042+0000","last_peered":"2024-02-10T05:55:35.741042+0000","last_clean":"2024-02-10T05:55:35.741042+0000","last_became_active":"2024-02-10T05:55:02.584812+0000","last_became_peered":"2024-02-10T05:55:02.584812+0000","last_unstale":"2024-02-10T05:55:35.741042+0000","last_undegraded":"2024-02-10T05:55:35.741042+0000","last_fullsized":"2024-02-10T05:55:35.741042+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:31:21.797773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:56:06.597580+0000","last_change":"2024-02-10T05:55:00.572045+0000","last_active":"2024-02-10T05:56:06.597580+0000","last_peered":"2024-02-10T05:56:06.597580+0000","last_clean":"2024-02-10T05:56:06.597580+0000","last_became_active":"2024-02-10T05:55:00.571758+0000","last_became_peered":"2024-02-10T05:55:00.571758+0000","last_unstale":"2024-02-10T05:56:06.597580+0000","last_undegraded":"2024-02-10T05:56:06.597580+0000","last_fullsized":"2024-02-10T05:56:06.597580+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:57:10.810245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.031196+0000","last_change":"2024-02-10T05:54:56.542343+0000","last_active":"2024-02-10T05:55:36.031196+0000","last_peered":"2024-02-10T05:55:36.031196+0000","last_clean":"2024-02-10T05:55:36.031196+0000","last_became_active":"2024-02-10T05:54:56.541409+0000","last_became_peered":"2024-02-10T05:54:56.541409+0000","last_unstale":"2024-02-10T05:55:36.031196+0000","last_undegraded":"2024-02-10T05:55:36.031196+0000","last_fullsized":"2024-02-10T05:55:36.031196+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:45:17.512782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"67'22","reported_seq":68,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338553+0000","last_change":"2024-02-10T05:54:58.561627+0000","last_active":"2024-02-10T05:55:36.338553+0000","last_peered":"2024-02-10T05:55:36.338553+0000","last_clean":"2024-02-10T05:55:36.338553+0000","last_became_active":"2024-02-10T05:54:58.560947+0000","last_became_peered":"2024-02-10T05:54:58.560947+0000","last_unstale":"2024-02-10T05:55:36.338553+0000","last_undegraded":"2024-02-10T05:55:36.338553+0000","last_fullsized":"2024-02-10T05:55:36.338553+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:03:16.214090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.736472+0000","last_change":"2024-02-10T05:55:02.581266+0000","last_active":"2024-02-10T05:55:35.736472+0000","last_peered":"2024-02-10T05:55:35.736472+0000","last_clean":"2024-02-10T05:55:35.736472+0000","last_became_active":"2024-02-10T05:55:02.580728+0000","last_became_peered":"2024-02-10T05:55:02.580728+0000","last_unstale":"2024-02-10T05:55:35.736472+0000","last_undegraded":"2024-02-10T05:55:35.736472+0000","last_fullsized":"2024-02-10T05:55:35.736472+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:22:22.437287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:56:06.597437+0000","last_change":"2024-02-10T05:55:00.572209+0000","last_active":"2024-02-10T05:56:06.597437+0000","last_peered":"2024-02-10T05:56:06.597437+0000","last_clean":"2024-02-10T05:56:06.597437+0000","last_became_active":"2024-02-10T05:55:00.571781+0000","last_became_peered":"2024-02-10T05:55:00.571781+0000","last_unstale":"2024-02-10T05:56:06.597437+0000","last_undegraded":"2024-02-10T05:56:06.597437+0000","last_fullsized":"2024-02-10T05:56:06.597437+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:02:56.718822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.739527+0000","last_change":"2024-02-10T05:54:56.543080+0000","last_active":"2024-02-10T05:55:35.739527+0000","last_peered":"2024-02-10T05:55:35.739527+0000","last_clean":"2024-02-10T05:55:35.739527+0000","last_became_active":"2024-02-10T05:54:56.542777+0000","last_became_peered":"2024-02-10T05:54:56.542777+0000","last_unstale":"2024-02-10T05:55:35.739527+0000","last_undegraded":"2024-02-10T05:55:35.739527+0000","last_fullsized":"2024-02-10T05:55:35.739527+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:38:59.251107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'15","reported_seq":57,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014876+0000","last_change":"2024-02-10T05:54:58.557620+0000","last_active":"2024-02-10T05:55:36.014876+0000","last_peered":"2024-02-10T05:55:36.014876+0000","last_clean":"2024-02-10T05:55:36.014876+0000","last_became_active":"2024-02-10T05:54:58.557283+0000","last_became_peered":"2024-02-10T05:54:58.557283+0000","last_unstale":"2024-02-10T05:55:36.014876+0000","last_undegraded":"2024-02-10T05:55:36.014876+0000","last_fullsized":"2024-02-10T05:55:36.014876+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:33:21.965789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.027640+0000","last_change":"2024-02-10T05:55:02.579787+0000","last_active":"2024-02-10T05:55:36.027640+0000","last_peered":"2024-02-10T05:55:36.027640+0000","last_clean":"2024-02-10T05:55:36.027640+0000","last_became_active":"2024-02-10T05:55:02.579393+0000","last_became_peered":"2024-02-10T05:55:02.579393+0000","last_unstale":"2024-02-10T05:55:36.027640+0000","last_undegraded":"2024-02-10T05:55:36.027640+0000","last_fullsized":"2024-02-10T05:55:36.027640+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:39:08.433187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.339117+0000","last_change":"2024-02-10T05:55:00.577404+0000","last_active":"2024-02-10T05:55:36.339117+0000","last_peered":"2024-02-10T05:55:36.339117+0000","last_clean":"2024-02-10T05:55:36.339117+0000","last_became_active":"2024-02-10T05:55:00.577117+0000","last_became_peered":"2024-02-10T05:55:00.577117+0000","last_unstale":"2024-02-10T05:55:36.339117+0000","last_undegraded":"2024-02-10T05:55:36.339117+0000","last_fullsized":"2024-02-10T05:55:36.339117+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:48:02.229013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.030865+0000","last_change":"2024-02-10T05:54:56.536012+0000","last_active":"2024-02-10T05:55:36.030865+0000","last_peered":"2024-02-10T05:55:36.030865+0000","last_clean":"2024-02-10T05:55:36.030865+0000","last_became_active":"2024-02-10T05:54:56.535738+0000","last_became_peered":"2024-02-10T05:54:56.535738+0000","last_unstale":"2024-02-10T05:55:36.030865+0000","last_undegraded":"2024-02-10T05:55:36.030865+0000","last_fullsized":"2024-02-10T05:55:36.030865+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:10:33.020334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'21","reported_seq":69,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.741937+0000","last_change":"2024-02-10T05:54:58.556634+0000","last_active":"2024-02-10T05:55:35.741937+0000","last_peered":"2024-02-10T05:55:35.741937+0000","last_clean":"2024-02-10T05:55:35.741937+0000","last_became_active":"2024-02-10T05:54:58.555438+0000","last_became_peered":"2024-02-10T05:54:58.555438+0000","last_unstale":"2024-02-10T05:55:35.741937+0000","last_undegraded":"2024-02-10T05:55:35.741937+0000","last_fullsized":"2024-02-10T05:55:35.741937+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:01:35.640408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.031308+0000","last_change":"2024-02-10T05:55:03.130669+0000","last_active":"2024-02-10T05:55:36.031308+0000","last_peered":"2024-02-10T05:55:36.031308+0000","last_clean":"2024-02-10T05:55:36.031308+0000","last_became_active":"2024-02-10T05:55:03.130187+0000","last_became_peered":"2024-02-10T05:55:03.130187+0000","last_unstale":"2024-02-10T05:55:36.031308+0000","last_undegraded":"2024-02-10T05:55:36.031308+0000","last_fullsized":"2024-02-10T05:55:36.031308+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:30:02.853983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014858+0000","last_change":"2024-02-10T05:55:00.572961+0000","last_active":"2024-02-10T05:55:36.014858+0000","last_peered":"2024-02-10T05:55:36.014858+0000","last_clean":"2024-02-10T05:55:36.014858+0000","last_became_active":"2024-02-10T05:55:00.572419+0000","last_became_peered":"2024-02-10T05:55:00.572419+0000","last_unstale":"2024-02-10T05:55:36.014858+0000","last_undegraded":"2024-02-10T05:55:36.014858+0000","last_fullsized":"2024-02-10T05:55:36.014858+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:52:42.978728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"60'1","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.337972+0000","last_change":"2024-02-10T05:54:56.538297+0000","last_active":"2024-02-10T05:55:36.337972+0000","last_peered":"2024-02-10T05:55:36.337972+0000","last_clean":"2024-02-10T05:55:36.337972+0000","last_became_active":"2024-02-10T05:54:56.538037+0000","last_became_peered":"2024-02-10T05:54:56.538037+0000","last_unstale":"2024-02-10T05:55:36.337972+0000","last_undegraded":"2024-02-10T05:55:36.337972+0000","last_fullsized":"2024-02-10T05:55:36.337972+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:34:40.947680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'12","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.339167+0000","last_change":"2024-02-10T05:54:58.561576+0000","last_active":"2024-02-10T05:55:36.339167+0000","last_peered":"2024-02-10T05:55:36.339167+0000","last_clean":"2024-02-10T05:55:36.339167+0000","last_became_active":"2024-02-10T05:54:58.560880+0000","last_became_peered":"2024-02-10T05:54:58.560880+0000","last_unstale":"2024-02-10T05:55:36.339167+0000","last_undegraded":"2024-02-10T05:55:36.339167+0000","last_fullsized":"2024-02-10T05:55:36.339167+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:16:11.344714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014511+0000","last_change":"2024-02-10T05:55:03.129351+0000","last_active":"2024-02-10T05:55:36.014511+0000","last_peered":"2024-02-10T05:55:36.014511+0000","last_clean":"2024-02-10T05:55:36.014511+0000","last_became_active":"2024-02-10T05:55:03.129054+0000","last_became_peered":"2024-02-10T05:55:03.129054+0000","last_unstale":"2024-02-10T05:55:36.014511+0000","last_undegraded":"2024-02-10T05:55:36.014511+0000","last_fullsized":"2024-02-10T05:55:36.014511+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:26:48.363844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.031585+0000","last_change":"2024-02-10T05:54:56.540743+0000","last_active":"2024-02-10T05:55:36.031585+0000","last_peered":"2024-02-10T05:55:36.031585+0000","last_clean":"2024-02-10T05:55:36.031585+0000","last_became_active":"2024-02-10T05:54:56.540470+0000","last_became_peered":"2024-02-10T05:54:56.540470+0000","last_unstale":"2024-02-10T05:55:36.031585+0000","last_undegraded":"2024-02-10T05:55:36.031585+0000","last_fullsized":"2024-02-10T05:55:36.031585+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:23:51.723375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"67'27","reported_seq":78,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014233+0000","last_change":"2024-02-10T05:54:58.557655+0000","last_active":"2024-02-10T05:55:36.014233+0000","last_peered":"2024-02-10T05:55:36.014233+0000","last_clean":"2024-02-10T05:55:36.014233+0000","last_became_active":"2024-02-10T05:54:58.557353+0000","last_became_peered":"2024-02-10T05:54:58.557353+0000","last_unstale":"2024-02-10T05:55:36.014233+0000","last_undegraded":"2024-02-10T05:55:36.014233+0000","last_fullsized":"2024-02-10T05:55:36.014233+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:53:54.536143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:56:06.597602+0000","last_change":"2024-02-10T05:55:00.577150+0000","last_active":"2024-02-10T05:56:06.597602+0000","last_peered":"2024-02-10T05:56:06.597602+0000","last_clean":"2024-02-10T05:56:06.597602+0000","last_became_active":"2024-02-10T05:55:00.576801+0000","last_became_peered":"2024-02-10T05:55:00.576801+0000","last_unstale":"2024-02-10T05:56:06.597602+0000","last_undegraded":"2024-02-10T05:56:06.597602+0000","last_fullsized":"2024-02-10T05:56:06.597602+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:56:06.372334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338372+0000","last_change":"2024-02-10T05:55:02.587625+0000","last_active":"2024-02-10T05:55:36.338372+0000","last_peered":"2024-02-10T05:55:36.338372+0000","last_clean":"2024-02-10T05:55:36.338372+0000","last_became_active":"2024-02-10T05:55:02.587320+0000","last_became_peered":"2024-02-10T05:55:02.587320+0000","last_unstale":"2024-02-10T05:55:36.338372+0000","last_undegraded":"2024-02-10T05:55:36.338372+0000","last_fullsized":"2024-02-10T05:55:36.338372+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:19:15.357080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.031567+0000","last_change":"2024-02-10T05:54:56.542102+0000","last_active":"2024-02-10T05:55:36.031567+0000","last_peered":"2024-02-10T05:55:36.031567+0000","last_clean":"2024-02-10T05:55:36.031567+0000","last_became_active":"2024-02-10T05:54:56.540632+0000","last_became_peered":"2024-02-10T05:54:56.540632+0000","last_unstale":"2024-02-10T05:55:36.031567+0000","last_undegraded":"2024-02-10T05:55:36.031567+0000","last_fullsized":"2024-02-10T05:55:36.031567+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:11:13.084896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"67'24","reported_seq":76,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.736682+0000","last_change":"2024-02-10T05:54:58.563244+0000","last_active":"2024-02-10T05:55:35.736682+0000","last_peered":"2024-02-10T05:55:35.736682+0000","last_clean":"2024-02-10T05:55:35.736682+0000","last_became_active":"2024-02-10T05:54:58.562959+0000","last_became_peered":"2024-02-10T05:54:58.562959+0000","last_unstale":"2024-02-10T05:55:35.736682+0000","last_undegraded":"2024-02-10T05:55:35.736682+0000","last_fullsized":"2024-02-10T05:55:35.736682+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:13:23.144091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.026823+0000","last_change":"2024-02-10T05:55:00.568890+0000","last_active":"2024-02-10T05:55:36.026823+0000","last_peered":"2024-02-10T05:55:36.026823+0000","last_clean":"2024-02-10T05:55:36.026823+0000","last_became_active":"2024-02-10T05:55:00.568490+0000","last_became_peered":"2024-02-10T05:55:00.568490+0000","last_unstale":"2024-02-10T05:55:36.026823+0000","last_undegraded":"2024-02-10T05:55:36.026823+0000","last_fullsized":"2024-02-10T05:55:36.026823+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:25:27.878666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.030788+0000","last_change":"2024-02-10T05:55:02.583935+0000","last_active":"2024-02-10T05:55:36.030788+0000","last_peered":"2024-02-10T05:55:36.030788+0000","last_clean":"2024-02-10T05:55:36.030788+0000","last_became_active":"2024-02-10T05:55:02.583552+0000","last_became_peered":"2024-02-10T05:55:02.583552+0000","last_unstale":"2024-02-10T05:55:36.030788+0000","last_undegraded":"2024-02-10T05:55:36.030788+0000","last_fullsized":"2024-02-10T05:55:36.030788+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:12:25.367129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.027312+0000","last_change":"2024-02-10T05:55:00.573921+0000","last_active":"2024-02-10T05:55:36.027312+0000","last_peered":"2024-02-10T05:55:36.027312+0000","last_clean":"2024-02-10T05:55:36.027312+0000","last_became_active":"2024-02-10T05:55:00.573600+0000","last_became_peered":"2024-02-10T05:55:00.573600+0000","last_unstale":"2024-02-10T05:55:36.027312+0000","last_undegraded":"2024-02-10T05:55:36.027312+0000","last_fullsized":"2024-02-10T05:55:36.027312+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:56:48.647371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.031561+0000","last_change":"2024-02-10T05:54:56.541434+0000","last_active":"2024-02-10T05:55:36.031561+0000","last_peered":"2024-02-10T05:55:36.031561+0000","last_clean":"2024-02-10T05:55:36.031561+0000","last_became_active":"2024-02-10T05:54:56.541103+0000","last_became_peered":"2024-02-10T05:54:56.541103+0000","last_unstale":"2024-02-10T05:55:36.031561+0000","last_undegraded":"2024-02-10T05:55:36.031561+0000","last_fullsized":"2024-02-10T05:55:36.031561+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:28:11.293509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"67'25","reported_seq":72,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338322+0000","last_change":"2024-02-10T05:54:58.560683+0000","last_active":"2024-02-10T05:55:36.338322+0000","last_peered":"2024-02-10T05:55:36.338322+0000","last_clean":"2024-02-10T05:55:36.338322+0000","last_became_active":"2024-02-10T05:54:58.560254+0000","last_became_peered":"2024-02-10T05:54:58.560254+0000","last_unstale":"2024-02-10T05:55:36.338322+0000","last_undegraded":"2024-02-10T05:55:36.338322+0000","last_fullsized":"2024-02-10T05:55:36.338322+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:44:10.134562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.738990+0000","last_change":"2024-02-10T05:55:02.582079+0000","last_active":"2024-02-10T05:55:35.738990+0000","last_peered":"2024-02-10T05:55:35.738990+0000","last_clean":"2024-02-10T05:55:35.738990+0000","last_became_active":"2024-02-10T05:55:02.581653+0000","last_became_peered":"2024-02-10T05:55:02.581653+0000","last_unstale":"2024-02-10T05:55:35.738990+0000","last_undegraded":"2024-02-10T05:55:35.738990+0000","last_fullsized":"2024-02-10T05:55:35.738990+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:13:58.090416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.027133+0000","last_change":"2024-02-10T05:55:00.576963+0000","last_active":"2024-02-10T05:55:36.027133+0000","last_peered":"2024-02-10T05:55:36.027133+0000","last_clean":"2024-02-10T05:55:36.027133+0000","last_became_active":"2024-02-10T05:55:00.576597+0000","last_became_peered":"2024-02-10T05:55:00.576597+0000","last_unstale":"2024-02-10T05:55:36.027133+0000","last_undegraded":"2024-02-10T05:55:36.027133+0000","last_fullsized":"2024-02-10T05:55:36.027133+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:41:11.237616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014603+0000","last_change":"2024-02-10T05:54:56.544756+0000","last_active":"2024-02-10T05:55:36.014603+0000","last_peered":"2024-02-10T05:55:36.014603+0000","last_clean":"2024-02-10T05:55:36.014603+0000","last_became_active":"2024-02-10T05:54:56.544460+0000","last_became_peered":"2024-02-10T05:54:56.544460+0000","last_unstale":"2024-02-10T05:55:36.014603+0000","last_undegraded":"2024-02-10T05:55:36.014603+0000","last_fullsized":"2024-02-10T05:55:36.014603+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:13:47.647366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"67'15","reported_seq":75,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.744406+0000","last_change":"2024-02-10T05:55:25.927808+0000","last_active":"2024-02-10T05:55:35.744406+0000","last_peered":"2024-02-10T05:55:35.744406+0000","last_clean":"2024-02-10T05:55:35.744406+0000","last_became_active":"2024-02-10T05:55:25.920369+0000","last_became_peered":"2024-02-10T05:55:25.920369+0000","last_unstale":"2024-02-10T05:55:35.744406+0000","last_undegraded":"2024-02-10T05:55:35.744406+0000","last_fullsized":"2024-02-10T05:55:35.744406+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:41:54.970006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338864+0000","last_change":"2024-02-10T05:55:02.581776+0000","last_active":"2024-02-10T05:55:36.338864+0000","last_peered":"2024-02-10T05:55:36.338864+0000","last_clean":"2024-02-10T05:55:36.338864+0000","last_became_active":"2024-02-10T05:55:02.581459+0000","last_became_peered":"2024-02-10T05:55:02.581459+0000","last_unstale":"2024-02-10T05:55:36.338864+0000","last_undegraded":"2024-02-10T05:55:36.338864+0000","last_fullsized":"2024-02-10T05:55:36.338864+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:34:16.027593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":42,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014361+0000","last_change":"2024-02-10T05:55:00.569960+0000","last_active":"2024-02-10T05:55:36.014361+0000","last_peered":"2024-02-10T05:55:36.014361+0000","last_clean":"2024-02-10T05:55:36.014361+0000","last_became_active":"2024-02-10T05:54:56.543145+0000","last_became_peered":"2024-02-10T05:54:56.543145+0000","last_unstale":"2024-02-10T05:55:36.014361+0000","last_undegraded":"2024-02-10T05:55:36.014361+0000","last_fullsized":"2024-02-10T05:55:36.014361+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:27:59.812610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047011520000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735203+0000","last_change":"2024-02-10T05:54:56.533400+0000","last_active":"2024-02-10T05:55:35.735203+0000","last_peered":"2024-02-10T05:55:35.735203+0000","last_clean":"2024-02-10T05:55:35.735203+0000","last_became_active":"2024-02-10T05:54:56.533110+0000","last_became_peered":"2024-02-10T05:54:56.533110+0000","last_unstale":"2024-02-10T05:55:35.735203+0000","last_undegraded":"2024-02-10T05:55:35.735203+0000","last_fullsized":"2024-02-10T05:55:35.735203+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:13:05.620672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"67'34","reported_seq":105,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:56:06.597383+0000","last_change":"2024-02-10T05:54:58.557933+0000","last_active":"2024-02-10T05:56:06.597383+0000","last_peered":"2024-02-10T05:56:06.597383+0000","last_clean":"2024-02-10T05:56:06.597383+0000","last_became_active":"2024-02-10T05:54:58.557563+0000","last_became_peered":"2024-02-10T05:54:58.557563+0000","last_unstale":"2024-02-10T05:56:06.597383+0000","last_undegraded":"2024-02-10T05:56:06.597383+0000","last_fullsized":"2024-02-10T05:56:06.597383+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:33:59.536452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.739962+0000","last_change":"2024-02-10T05:55:00.563519+0000","last_active":"2024-02-10T05:55:35.739962+0000","last_peered":"2024-02-10T05:55:35.739962+0000","last_clean":"2024-02-10T05:55:35.739962+0000","last_became_active":"2024-02-10T05:55:00.562557+0000","last_became_peered":"2024-02-10T05:55:00.562557+0000","last_unstale":"2024-02-10T05:55:35.739962+0000","last_undegraded":"2024-02-10T05:55:35.739962+0000","last_fullsized":"2024-02-10T05:55:35.739962+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:22:44.771980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.031638+0000","last_change":"2024-02-10T05:55:03.130189+0000","last_active":"2024-02-10T05:55:36.031638+0000","last_peered":"2024-02-10T05:55:36.031638+0000","last_clean":"2024-02-10T05:55:36.031638+0000","last_became_active":"2024-02-10T05:55:03.129911+0000","last_became_peered":"2024-02-10T05:55:03.129911+0000","last_unstale":"2024-02-10T05:55:36.031638+0000","last_undegraded":"2024-02-10T05:55:36.031638+0000","last_fullsized":"2024-02-10T05:55:36.031638+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:57:22.598979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.742524+0000","last_change":"2024-02-10T05:54:56.542933+0000","last_active":"2024-02-10T05:55:35.742524+0000","last_peered":"2024-02-10T05:55:35.742524+0000","last_clean":"2024-02-10T05:55:35.742524+0000","last_became_active":"2024-02-10T05:54:56.542659+0000","last_became_peered":"2024-02-10T05:54:56.542659+0000","last_unstale":"2024-02-10T05:55:35.742524+0000","last_undegraded":"2024-02-10T05:55:35.742524+0000","last_fullsized":"2024-02-10T05:55:35.742524+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:05:37.975285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.026992+0000","last_change":"2024-02-10T05:55:00.564438+0000","last_active":"2024-02-10T05:55:36.026992+0000","last_peered":"2024-02-10T05:55:36.026992+0000","last_clean":"2024-02-10T05:55:36.026992+0000","last_became_active":"2024-02-10T05:54:56.544669+0000","last_became_peered":"2024-02-10T05:54:56.544669+0000","last_unstale":"2024-02-10T05:55:36.026992+0000","last_undegraded":"2024-02-10T05:55:36.026992+0000","last_fullsized":"2024-02-10T05:55:36.026992+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:37:25.584726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012041790000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'19","reported_seq":27,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.027523+0000","last_change":"2024-02-10T05:55:25.931284+0000","last_active":"2024-02-10T05:55:36.027523+0000","last_peered":"2024-02-10T05:55:36.027523+0000","last_clean":"2024-02-10T05:55:36.027523+0000","last_became_active":"2024-02-10T05:55:25.921942+0000","last_became_peered":"2024-02-10T05:55:25.921942+0000","last_unstale":"2024-02-10T05:55:36.027523+0000","last_undegraded":"2024-02-10T05:55:36.027523+0000","last_fullsized":"2024-02-10T05:55:36.027523+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:12:33.977505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":26,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.027482+0000","last_change":"2024-02-10T05:55:00.572364+0000","last_active":"2024-02-10T05:55:36.027482+0000","last_peered":"2024-02-10T05:55:36.027482+0000","last_clean":"2024-02-10T05:55:36.027482+0000","last_became_active":"2024-02-10T05:55:00.571963+0000","last_became_peered":"2024-02-10T05:55:00.571963+0000","last_unstale":"2024-02-10T05:55:36.027482+0000","last_undegraded":"2024-02-10T05:55:36.027482+0000","last_fullsized":"2024-02-10T05:55:36.027482+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:13:12.833829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014859+0000","last_change":"2024-02-10T05:55:02.582765+0000","last_active":"2024-02-10T05:55:36.014859+0000","last_peered":"2024-02-10T05:55:36.014859+0000","last_clean":"2024-02-10T05:55:36.014859+0000","last_became_active":"2024-02-10T05:55:02.582256+0000","last_became_peered":"2024-02-10T05:55:02.582256+0000","last_unstale":"2024-02-10T05:55:36.014859+0000","last_undegraded":"2024-02-10T05:55:36.014859+0000","last_fullsized":"2024-02-10T05:55:36.014859+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:24:07.550336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"71'188","reported_seq":345,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:37.963163+0000","last_change":"2024-02-10T05:54:47.640507+0000","last_active":"2024-02-10T05:55:37.963163+0000","last_peered":"2024-02-10T05:55:37.963163+0000","last_clean":"2024-02-10T05:55:37.963163+0000","last_became_active":"2024-02-10T05:54:47.640071+0000","last_became_peered":"2024-02-10T05:54:47.640071+0000","last_unstale":"2024-02-10T05:55:37.963163+0000","last_undegraded":"2024-02-10T05:55:37.963163+0000","last_fullsized":"2024-02-10T05:55:37.963163+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:52:38.887992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:52:38.887992+0000","last_clean_scrub_stamp":"2024-02-10T05:52:38.887992+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:16:51.052806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338294+0000","last_change":"2024-02-10T05:54:56.543222+0000","last_active":"2024-02-10T05:55:36.338294+0000","last_peered":"2024-02-10T05:55:36.338294+0000","last_clean":"2024-02-10T05:55:36.338294+0000","last_became_active":"2024-02-10T05:54:56.542949+0000","last_became_peered":"2024-02-10T05:54:56.542949+0000","last_unstale":"2024-02-10T05:55:36.338294+0000","last_undegraded":"2024-02-10T05:55:36.338294+0000","last_fullsized":"2024-02-10T05:55:36.338294+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:18:35.117143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"67'16","reported_seq":67,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:56:06.597602+0000","last_change":"2024-02-10T05:54:58.562821+0000","last_active":"2024-02-10T05:56:06.597602+0000","last_peered":"2024-02-10T05:56:06.597602+0000","last_clean":"2024-02-10T05:56:06.597602+0000","last_became_active":"2024-02-10T05:54:58.562517+0000","last_became_peered":"2024-02-10T05:54:58.562517+0000","last_unstale":"2024-02-10T05:56:06.597602+0000","last_undegraded":"2024-02-10T05:56:06.597602+0000","last_fullsized":"2024-02-10T05:56:06.597602+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:38:12.263776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.031373+0000","last_change":"2024-02-10T05:55:00.574325+0000","last_active":"2024-02-10T05:55:36.031373+0000","last_peered":"2024-02-10T05:55:36.031373+0000","last_clean":"2024-02-10T05:55:36.031373+0000","last_became_active":"2024-02-10T05:55:00.574036+0000","last_became_peered":"2024-02-10T05:55:00.574036+0000","last_unstale":"2024-02-10T05:55:36.031373+0000","last_undegraded":"2024-02-10T05:55:36.031373+0000","last_fullsized":"2024-02-10T05:55:36.031373+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:28:14.174664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.743650+0000","last_change":"2024-02-10T05:55:02.586999+0000","last_active":"2024-02-10T05:55:35.743650+0000","last_peered":"2024-02-10T05:55:35.743650+0000","last_clean":"2024-02-10T05:55:35.743650+0000","last_became_active":"2024-02-10T05:55:02.586660+0000","last_became_peered":"2024-02-10T05:55:02.586660+0000","last_unstale":"2024-02-10T05:55:35.743650+0000","last_undegraded":"2024-02-10T05:55:35.743650+0000","last_fullsized":"2024-02-10T05:55:35.743650+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:17:05.193264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.739223+0000","last_change":"2024-02-10T05:54:56.544656+0000","last_active":"2024-02-10T05:55:35.739223+0000","last_peered":"2024-02-10T05:55:35.739223+0000","last_clean":"2024-02-10T05:55:35.739223+0000","last_became_active":"2024-02-10T05:54:56.544382+0000","last_became_peered":"2024-02-10T05:54:56.544382+0000","last_unstale":"2024-02-10T05:55:35.739223+0000","last_undegraded":"2024-02-10T05:55:35.739223+0000","last_fullsized":"2024-02-10T05:55:35.739223+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:16:42.905086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":112,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:56:10.551619+0000","last_change":"2024-02-10T05:55:00.566727+0000","last_active":"2024-02-10T05:56:10.551619+0000","last_peered":"2024-02-10T05:56:10.551619+0000","last_clean":"2024-02-10T05:56:10.551619+0000","last_became_active":"2024-02-10T05:54:56.539786+0000","last_became_peered":"2024-02-10T05:54:56.539786+0000","last_unstale":"2024-02-10T05:56:10.551619+0000","last_undegraded":"2024-02-10T05:56:10.551619+0000","last_fullsized":"2024-02-10T05:56:10.551619+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:17:41.995852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001442444,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":64,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":47,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.741150+0000","last_change":"2024-02-10T05:54:58.557165+0000","last_active":"2024-02-10T05:55:35.741150+0000","last_peered":"2024-02-10T05:55:35.741150+0000","last_clean":"2024-02-10T05:55:35.741150+0000","last_became_active":"2024-02-10T05:54:58.556344+0000","last_became_peered":"2024-02-10T05:54:58.556344+0000","last_unstale":"2024-02-10T05:55:35.741150+0000","last_undegraded":"2024-02-10T05:55:35.741150+0000","last_fullsized":"2024-02-10T05:55:35.741150+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:12:08.664273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.015042+0000","last_change":"2024-02-10T05:55:00.578052+0000","last_active":"2024-02-10T05:55:36.015042+0000","last_peered":"2024-02-10T05:55:36.015042+0000","last_clean":"2024-02-10T05:55:36.015042+0000","last_became_active":"2024-02-10T05:55:00.577752+0000","last_became_peered":"2024-02-10T05:55:00.577752+0000","last_unstale":"2024-02-10T05:55:36.015042+0000","last_undegraded":"2024-02-10T05:55:36.015042+0000","last_fullsized":"2024-02-10T05:55:36.015042+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:59:14.813996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.745494+0000","last_change":"2024-02-10T05:55:03.129468+0000","last_active":"2024-02-10T05:55:35.745494+0000","last_peered":"2024-02-10T05:55:35.745494+0000","last_clean":"2024-02-10T05:55:35.745494+0000","last_became_active":"2024-02-10T05:55:03.129074+0000","last_became_peered":"2024-02-10T05:55:03.129074+0000","last_unstale":"2024-02-10T05:55:35.745494+0000","last_undegraded":"2024-02-10T05:55:35.745494+0000","last_fullsized":"2024-02-10T05:55:35.745494+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:55:37.794146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.745373+0000","last_change":"2024-02-10T05:54:56.541048+0000","last_active":"2024-02-10T05:55:35.745373+0000","last_peered":"2024-02-10T05:55:35.745373+0000","last_clean":"2024-02-10T05:55:35.745373+0000","last_became_active":"2024-02-10T05:54:56.540749+0000","last_became_peered":"2024-02-10T05:54:56.540749+0000","last_unstale":"2024-02-10T05:55:35.745373+0000","last_undegraded":"2024-02-10T05:55:35.745373+0000","last_fullsized":"2024-02-10T05:55:35.745373+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:22:30.533260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"67'25","reported_seq":77,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.744112+0000","last_change":"2024-02-10T05:54:58.557311+0000","last_active":"2024-02-10T05:55:35.744112+0000","last_peered":"2024-02-10T05:55:35.744112+0000","last_clean":"2024-02-10T05:55:35.744112+0000","last_became_active":"2024-02-10T05:54:58.556986+0000","last_became_peered":"2024-02-10T05:54:58.556986+0000","last_unstale":"2024-02-10T05:55:35.744112+0000","last_undegraded":"2024-02-10T05:55:35.744112+0000","last_fullsized":"2024-02-10T05:55:35.744112+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:11:21.366052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338270+0000","last_change":"2024-02-10T05:55:00.576171+0000","last_active":"2024-02-10T05:55:36.338270+0000","last_peered":"2024-02-10T05:55:36.338270+0000","last_clean":"2024-02-10T05:55:36.338270+0000","last_became_active":"2024-02-10T05:55:00.575793+0000","last_became_peered":"2024-02-10T05:55:00.575793+0000","last_unstale":"2024-02-10T05:55:36.338270+0000","last_undegraded":"2024-02-10T05:55:36.338270+0000","last_fullsized":"2024-02-10T05:55:36.338270+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:33:05.098656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.738607+0000","last_change":"2024-02-10T05:55:02.584717+0000","last_active":"2024-02-10T05:55:35.738607+0000","last_peered":"2024-02-10T05:55:35.738607+0000","last_clean":"2024-02-10T05:55:35.738607+0000","last_became_active":"2024-02-10T05:55:02.584406+0000","last_became_peered":"2024-02-10T05:55:02.584406+0000","last_unstale":"2024-02-10T05:55:35.738607+0000","last_undegraded":"2024-02-10T05:55:35.738607+0000","last_fullsized":"2024-02-10T05:55:35.738607+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:05:36.016999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014722+0000","last_change":"2024-02-10T05:54:56.545279+0000","last_active":"2024-02-10T05:55:36.014722+0000","last_peered":"2024-02-10T05:55:36.014722+0000","last_clean":"2024-02-10T05:55:36.014722+0000","last_became_active":"2024-02-10T05:54:56.545023+0000","last_became_peered":"2024-02-10T05:54:56.545023+0000","last_unstale":"2024-02-10T05:55:36.014722+0000","last_undegraded":"2024-02-10T05:55:36.014722+0000","last_fullsized":"2024-02-10T05:55:36.014722+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:26:36.285352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"67'18","reported_seq":58,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.743139+0000","last_change":"2024-02-10T05:54:58.566635+0000","last_active":"2024-02-10T05:55:35.743139+0000","last_peered":"2024-02-10T05:55:35.743139+0000","last_clean":"2024-02-10T05:55:35.743139+0000","last_became_active":"2024-02-10T05:54:58.565164+0000","last_became_peered":"2024-02-10T05:54:58.565164+0000","last_unstale":"2024-02-10T05:55:35.743139+0000","last_undegraded":"2024-02-10T05:55:35.743139+0000","last_fullsized":"2024-02-10T05:55:35.743139+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:41:48.201138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:56:06.597371+0000","last_change":"2024-02-10T05:55:00.573505+0000","last_active":"2024-02-10T05:56:06.597371+0000","last_peered":"2024-02-10T05:56:06.597371+0000","last_clean":"2024-02-10T05:56:06.597371+0000","last_became_active":"2024-02-10T05:55:00.573222+0000","last_became_peered":"2024-02-10T05:55:00.573222+0000","last_unstale":"2024-02-10T05:56:06.597371+0000","last_undegraded":"2024-02-10T05:56:06.597371+0000","last_fullsized":"2024-02-10T05:56:06.597371+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:30:37.386873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.031370+0000","last_change":"2024-02-10T05:55:03.129638+0000","last_active":"2024-02-10T05:55:36.031370+0000","last_peered":"2024-02-10T05:55:36.031370+0000","last_clean":"2024-02-10T05:55:36.031370+0000","last_became_active":"2024-02-10T05:55:03.129290+0000","last_became_peered":"2024-02-10T05:55:03.129290+0000","last_unstale":"2024-02-10T05:55:36.031370+0000","last_undegraded":"2024-02-10T05:55:36.031370+0000","last_fullsized":"2024-02-10T05:55:36.031370+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:31:40.725143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.741295+0000","last_change":"2024-02-10T05:54:56.541481+0000","last_active":"2024-02-10T05:55:35.741295+0000","last_peered":"2024-02-10T05:55:35.741295+0000","last_clean":"2024-02-10T05:55:35.741295+0000","last_became_active":"2024-02-10T05:54:56.541170+0000","last_became_peered":"2024-02-10T05:54:56.541170+0000","last_unstale":"2024-02-10T05:55:35.741295+0000","last_undegraded":"2024-02-10T05:55:35.741295+0000","last_fullsized":"2024-02-10T05:55:35.741295+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:33:14.751742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"67'20","reported_seq":62,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.736547+0000","last_change":"2024-02-10T05:54:58.562058+0000","last_active":"2024-02-10T05:55:35.736547+0000","last_peered":"2024-02-10T05:55:35.736547+0000","last_clean":"2024-02-10T05:55:35.736547+0000","last_became_active":"2024-02-10T05:54:58.561057+0000","last_became_peered":"2024-02-10T05:54:58.561057+0000","last_unstale":"2024-02-10T05:55:35.736547+0000","last_undegraded":"2024-02-10T05:55:35.736547+0000","last_fullsized":"2024-02-10T05:55:35.736547+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:54:31.311039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:56:06.597417+0000","last_change":"2024-02-10T05:55:00.580085+0000","last_active":"2024-02-10T05:56:06.597417+0000","last_peered":"2024-02-10T05:56:06.597417+0000","last_clean":"2024-02-10T05:56:06.597417+0000","last_became_active":"2024-02-10T05:55:00.579395+0000","last_became_peered":"2024-02-10T05:55:00.579395+0000","last_unstale":"2024-02-10T05:56:06.597417+0000","last_undegraded":"2024-02-10T05:56:06.597417+0000","last_fullsized":"2024-02-10T05:56:06.597417+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:48:50.506985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735321+0000","last_change":"2024-02-10T05:55:02.584535+0000","last_active":"2024-02-10T05:55:35.735321+0000","last_peered":"2024-02-10T05:55:35.735321+0000","last_clean":"2024-02-10T05:55:35.735321+0000","last_became_active":"2024-02-10T05:55:02.583871+0000","last_became_peered":"2024-02-10T05:55:02.583871+0000","last_unstale":"2024-02-10T05:55:35.735321+0000","last_undegraded":"2024-02-10T05:55:35.735321+0000","last_fullsized":"2024-02-10T05:55:35.735321+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:54:30.587897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.339020+0000","last_change":"2024-02-10T05:54:56.543104+0000","last_active":"2024-02-10T05:55:36.339020+0000","last_peered":"2024-02-10T05:55:36.339020+0000","last_clean":"2024-02-10T05:55:36.339020+0000","last_became_active":"2024-02-10T05:54:56.542806+0000","last_became_peered":"2024-02-10T05:54:56.542806+0000","last_unstale":"2024-02-10T05:55:36.339020+0000","last_undegraded":"2024-02-10T05:55:36.339020+0000","last_fullsized":"2024-02-10T05:55:36.339020+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:12:06.292926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"67'30","reported_seq":85,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338143+0000","last_change":"2024-02-10T05:54:58.564802+0000","last_active":"2024-02-10T05:55:36.338143+0000","last_peered":"2024-02-10T05:55:36.338143+0000","last_clean":"2024-02-10T05:55:36.338143+0000","last_became_active":"2024-02-10T05:54:58.563860+0000","last_became_peered":"2024-02-10T05:54:58.563860+0000","last_unstale":"2024-02-10T05:55:36.338143+0000","last_undegraded":"2024-02-10T05:55:36.338143+0000","last_fullsized":"2024-02-10T05:55:36.338143+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:21:57.209980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735330+0000","last_change":"2024-02-10T05:55:00.579117+0000","last_active":"2024-02-10T05:55:35.735330+0000","last_peered":"2024-02-10T05:55:35.735330+0000","last_clean":"2024-02-10T05:55:35.735330+0000","last_became_active":"2024-02-10T05:55:00.578770+0000","last_became_peered":"2024-02-10T05:55:00.578770+0000","last_unstale":"2024-02-10T05:55:35.735330+0000","last_undegraded":"2024-02-10T05:55:35.735330+0000","last_fullsized":"2024-02-10T05:55:35.735330+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:35:37.992788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014575+0000","last_change":"2024-02-10T05:55:02.582138+0000","last_active":"2024-02-10T05:55:36.014575+0000","last_peered":"2024-02-10T05:55:36.014575+0000","last_clean":"2024-02-10T05:55:36.014575+0000","last_became_active":"2024-02-10T05:55:02.581817+0000","last_became_peered":"2024-02-10T05:55:02.581817+0000","last_unstale":"2024-02-10T05:55:36.014575+0000","last_undegraded":"2024-02-10T05:55:36.014575+0000","last_fullsized":"2024-02-10T05:55:36.014575+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:08:31.228173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:56:06.597628+0000","last_change":"2024-02-10T05:55:00.571661+0000","last_active":"2024-02-10T05:56:06.597628+0000","last_peered":"2024-02-10T05:56:06.597628+0000","last_clean":"2024-02-10T05:56:06.597628+0000","last_became_active":"2024-02-10T05:55:00.571303+0000","last_became_peered":"2024-02-10T05:55:00.571303+0000","last_unstale":"2024-02-10T05:56:06.597628+0000","last_undegraded":"2024-02-10T05:56:06.597628+0000","last_fullsized":"2024-02-10T05:56:06.597628+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:48:42.433150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338056+0000","last_change":"2024-02-10T05:54:56.542011+0000","last_active":"2024-02-10T05:55:36.338056+0000","last_peered":"2024-02-10T05:55:36.338056+0000","last_clean":"2024-02-10T05:55:36.338056+0000","last_became_active":"2024-02-10T05:54:56.541687+0000","last_became_peered":"2024-02-10T05:54:56.541687+0000","last_unstale":"2024-02-10T05:55:36.338056+0000","last_undegraded":"2024-02-10T05:55:36.338056+0000","last_fullsized":"2024-02-10T05:55:36.338056+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:55:43.359225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.745885+0000","last_change":"2024-02-10T05:54:58.558114+0000","last_active":"2024-02-10T05:55:35.745885+0000","last_peered":"2024-02-10T05:55:35.745885+0000","last_clean":"2024-02-10T05:55:35.745885+0000","last_became_active":"2024-02-10T05:54:58.557771+0000","last_became_peered":"2024-02-10T05:54:58.557771+0000","last_unstale":"2024-02-10T05:55:35.745885+0000","last_undegraded":"2024-02-10T05:55:35.745885+0000","last_fullsized":"2024-02-10T05:55:35.745885+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:03:19.565049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338454+0000","last_change":"2024-02-10T05:55:03.129563+0000","last_active":"2024-02-10T05:55:36.338454+0000","last_peered":"2024-02-10T05:55:36.338454+0000","last_clean":"2024-02-10T05:55:36.338454+0000","last_became_active":"2024-02-10T05:55:03.129023+0000","last_became_peered":"2024-02-10T05:55:03.129023+0000","last_unstale":"2024-02-10T05:55:36.338454+0000","last_undegraded":"2024-02-10T05:55:36.338454+0000","last_fullsized":"2024-02-10T05:55:36.338454+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:54:54.545671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014988+0000","last_change":"2024-02-10T05:55:00.580520+0000","last_active":"2024-02-10T05:55:36.014988+0000","last_peered":"2024-02-10T05:55:36.014988+0000","last_clean":"2024-02-10T05:55:36.014988+0000","last_became_active":"2024-02-10T05:55:00.580196+0000","last_became_peered":"2024-02-10T05:55:00.580196+0000","last_unstale":"2024-02-10T05:55:36.014988+0000","last_undegraded":"2024-02-10T05:55:36.014988+0000","last_fullsized":"2024-02-10T05:55:36.014988+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:49:08.651272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735153+0000","last_change":"2024-02-10T05:54:56.534306+0000","last_active":"2024-02-10T05:55:35.735153+0000","last_peered":"2024-02-10T05:55:35.735153+0000","last_clean":"2024-02-10T05:55:35.735153+0000","last_became_active":"2024-02-10T05:54:56.533672+0000","last_became_peered":"2024-02-10T05:54:56.533672+0000","last_unstale":"2024-02-10T05:55:35.735153+0000","last_undegraded":"2024-02-10T05:55:35.735153+0000","last_fullsized":"2024-02-10T05:55:35.735153+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:28:23.020328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"67'17","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735488+0000","last_change":"2024-02-10T05:54:58.563075+0000","last_active":"2024-02-10T05:55:35.735488+0000","last_peered":"2024-02-10T05:55:35.735488+0000","last_clean":"2024-02-10T05:55:35.735488+0000","last_became_active":"2024-02-10T05:54:58.562774+0000","last_became_peered":"2024-02-10T05:54:58.562774+0000","last_unstale":"2024-02-10T05:55:35.735488+0000","last_undegraded":"2024-02-10T05:55:35.735488+0000","last_fullsized":"2024-02-10T05:55:35.735488+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:00:33.210657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.027690+0000","last_change":"2024-02-10T05:55:02.583875+0000","last_active":"2024-02-10T05:55:36.027690+0000","last_peered":"2024-02-10T05:55:36.027690+0000","last_clean":"2024-02-10T05:55:36.027690+0000","last_became_active":"2024-02-10T05:55:02.583508+0000","last_became_peered":"2024-02-10T05:55:02.583508+0000","last_unstale":"2024-02-10T05:55:36.027690+0000","last_undegraded":"2024-02-10T05:55:36.027690+0000","last_fullsized":"2024-02-10T05:55:36.027690+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:23:35.028206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.745710+0000","last_change":"2024-02-10T05:55:00.570046+0000","last_active":"2024-02-10T05:55:35.745710+0000","last_peered":"2024-02-10T05:55:35.745710+0000","last_clean":"2024-02-10T05:55:35.745710+0000","last_became_active":"2024-02-10T05:55:00.569754+0000","last_became_peered":"2024-02-10T05:55:00.569754+0000","last_unstale":"2024-02-10T05:55:35.745710+0000","last_undegraded":"2024-02-10T05:55:35.745710+0000","last_fullsized":"2024-02-10T05:55:35.745710+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:09:55.813272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338108+0000","last_change":"2024-02-10T05:54:56.543143+0000","last_active":"2024-02-10T05:55:36.338108+0000","last_peered":"2024-02-10T05:55:36.338108+0000","last_clean":"2024-02-10T05:55:36.338108+0000","last_became_active":"2024-02-10T05:54:56.542841+0000","last_became_peered":"2024-02-10T05:54:56.542841+0000","last_unstale":"2024-02-10T05:55:36.338108+0000","last_undegraded":"2024-02-10T05:55:36.338108+0000","last_fullsized":"2024-02-10T05:55:36.338108+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:21:42.837802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":57,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.736194+0000","last_change":"2024-02-10T05:54:58.563683+0000","last_active":"2024-02-10T05:55:35.736194+0000","last_peered":"2024-02-10T05:55:35.736194+0000","last_clean":"2024-02-10T05:55:35.736194+0000","last_became_active":"2024-02-10T05:54:58.563288+0000","last_became_peered":"2024-02-10T05:54:58.563288+0000","last_unstale":"2024-02-10T05:55:35.736194+0000","last_undegraded":"2024-02-10T05:55:35.736194+0000","last_fullsized":"2024-02-10T05:55:35.736194+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:32:36.944544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.736016+0000","last_change":"2024-02-10T05:55:02.583004+0000","last_active":"2024-02-10T05:55:35.736016+0000","last_peered":"2024-02-10T05:55:35.736016+0000","last_clean":"2024-02-10T05:55:35.736016+0000","last_became_active":"2024-02-10T05:55:02.582694+0000","last_became_peered":"2024-02-10T05:55:02.582694+0000","last_unstale":"2024-02-10T05:55:35.736016+0000","last_undegraded":"2024-02-10T05:55:35.736016+0000","last_fullsized":"2024-02-10T05:55:35.736016+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:44:46.960181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":52,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:56:06.597359+0000","last_change":"2024-02-10T05:55:00.574688+0000","last_active":"2024-02-10T05:56:06.597359+0000","last_peered":"2024-02-10T05:56:06.597359+0000","last_clean":"2024-02-10T05:56:06.597359+0000","last_became_active":"2024-02-10T05:55:00.574385+0000","last_became_peered":"2024-02-10T05:55:00.574385+0000","last_unstale":"2024-02-10T05:56:06.597359+0000","last_undegraded":"2024-02-10T05:56:06.597359+0000","last_fullsized":"2024-02-10T05:56:06.597359+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:56:06.166451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338464+0000","last_change":"2024-02-10T05:54:56.549290+0000","last_active":"2024-02-10T05:55:36.338464+0000","last_peered":"2024-02-10T05:55:36.338464+0000","last_clean":"2024-02-10T05:55:36.338464+0000","last_became_active":"2024-02-10T05:54:56.548651+0000","last_became_peered":"2024-02-10T05:54:56.548651+0000","last_unstale":"2024-02-10T05:55:36.338464+0000","last_undegraded":"2024-02-10T05:55:36.338464+0000","last_fullsized":"2024-02-10T05:55:36.338464+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:28:41.180202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.736041+0000","last_change":"2024-02-10T05:54:58.565766+0000","last_active":"2024-02-10T05:55:35.736041+0000","last_peered":"2024-02-10T05:55:35.736041+0000","last_clean":"2024-02-10T05:55:35.736041+0000","last_became_active":"2024-02-10T05:54:58.565487+0000","last_became_peered":"2024-02-10T05:54:58.565487+0000","last_unstale":"2024-02-10T05:55:35.736041+0000","last_undegraded":"2024-02-10T05:55:35.736041+0000","last_fullsized":"2024-02-10T05:55:35.736041+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:30:40.248722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":985,"num_read_kb":710,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":587,"ondisk_log_size":587,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":188,"ondisk_log_size":188,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":64,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234004,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32540,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708516,"statfs":{"total":95990841344,"available":95957520384,"internally_reserved":0,"allocated":1396736,"data_stored":923975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.59199999999999997}]},{"osd":1,"last update":"Sat Feb 10 05:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.61499999999999999}]},{"osd":2,"last update":"Sat Feb 10 05:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.49099999999999999}]},{"osd":3,"last update":"Sat Feb 10 05:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.53900000000000003}]},{"osd":4,"last update":"Sat Feb 10 05:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.371}]},{"osd":5,"last update":"Sat Feb 10 05:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.56100000000000005}]},{"osd":6,"last update":"Sat Feb 10 05:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.505},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.46400000000000002}]}]},{"osd":6,"up_from":46,"seq":197568495641,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":28356,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93712700,"statfs":{"total":95990841344,"available":95961804800,"internally_reserved":0,"allocated":1372160,"data_stored":922435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.65000000000000002}]},{"osd":1,"last update":"Sat Feb 10 05:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.629}]},{"osd":2,"last update":"Sat Feb 10 05:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.55900000000000005}]},{"osd":3,"last update":"Sat Feb 10 05:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.53600000000000003}]},{"osd":4,"last update":"Sat Feb 10 05:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.752}]},{"osd":5,"last update":"Sat Feb 10 05:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.67600000000000005}]},{"osd":7,"last update":"Sat Feb 10 05:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.71599999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31980,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709076,"statfs":{"total":95990841344,"available":95958093824,"internally_reserved":0,"allocated":819200,"data_stored":332904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.33800000000000002,"15min":0.308},"min":{"1min":0.248,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.33900000000000002,"15min":0.31},"min":{"1min":0.246,"5min":0.223,"15min":0.223},"max":{"1min":0.56000000000000005,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.32200000000000001}]},{"osd":2,"last update":"Sat Feb 10 05:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.37,"15min":0.35999999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.58599999999999997,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.40100000000000002,"15min":0.38700000000000001},"min":{"1min":0.27000000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.38700000000000001}]},{"osd":3,"last update":"Sat Feb 10 05:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.46300000000000002,"15min":0.46700000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.60199999999999998,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.44,"5min":0.45300000000000001,"15min":0.45500000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.35899999999999999}]},{"osd":4,"last update":"Sat Feb 10 05:55:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.621,"15min":0.628},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.81699999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.77400000000000002,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.56999999999999995}]},{"osd":5,"last update":"Sat Feb 10 05:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.61499999999999999,"15min":0.61699999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.751,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57999999999999996,"15min":0.57699999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82899999999999996,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.60499999999999998}]},{"osd":6,"last update":"Sat Feb 10 05:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.623}]},{"osd":7,"last update":"Sat Feb 10 05:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.66000000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32544,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708512,"statfs":{"total":95990841344,"available":95957516288,"internally_reserved":0,"allocated":1400832,"data_stored":924103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 05:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.34899999999999998,"15min":0.32500000000000001},"min":{"1min":0.25600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.56200000000000006,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.36699999999999999,"15min":0.34599999999999997},"min":{"1min":0.24399999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.65700000000000003,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.375}]},{"osd":2,"last update":"Sat Feb 10 05:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.39000000000000001,"15min":0.36699999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.69899999999999995,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.38700000000000001,"15min":0.377},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.27100000000000002}]},{"osd":3,"last update":"Sat Feb 10 05:56:09 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.46500000000000002,"15min":0.46400000000000002},"min":{"1min":0.32600000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.45300000000000001,"15min":0.45600000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.59199999999999997,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.312}]},{"osd":4,"last update":"Sat Feb 10 05:55:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.60999999999999999,"15min":0.60599999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.83199999999999996,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.61799999999999999,"15min":0.61599999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.47199999999999998}]},{"osd":5,"last update":"Sat Feb 10 05:55:56 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.63700000000000001,"15min":0.63900000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.61599999999999999,"15min":0.61899999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.86199999999999999,"5min":0.879,"15min":0.879},"last":0.42199999999999999}]},{"osd":6,"last update":"Sat Feb 10 05:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.432},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.41199999999999998}]},{"osd":7,"last update":"Sat Feb 10 05:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.48699999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27792,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713264,"statfs":{"total":95990841344,"available":95962382336,"internally_reserved":0,"allocated":790528,"data_stored":332236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.40999999999999998,"15min":0.40200000000000002},"min":{"1min":0.26600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.61799999999999999,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57699999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.38600000000000001}]},{"osd":1,"last update":"Sat Feb 10 05:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.41999999999999998,"15min":0.42199999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.63700000000000001,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.41999999999999998,"15min":0.40999999999999998},"min":{"1min":0.26500000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.65400000000000003,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.34699999999999998}]},{"osd":3,"last update":"Sat Feb 10 05:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34999999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.501,"15min":0.499},"min":{"1min":0.35099999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.40699999999999997}]},{"osd":4,"last update":"Sat Feb 10 05:55:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58299999999999996,"15min":0.58199999999999996},"min":{"1min":0.41899999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.47799999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.82799999999999996}]},{"osd":5,"last update":"Sat Feb 10 05:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.64000000000000001,"15min":0.65000000000000002},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.78600000000000003,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.61099999999999999,"15min":0.622},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.73999999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.75900000000000001}]},{"osd":6,"last update":"Sat Feb 10 05:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.59999999999999998}]},{"osd":7,"last update":"Sat Feb 10 05:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.76900000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182440,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31976,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709080,"statfs":{"total":95990841344,"available":95958097920,"internally_reserved":0,"allocated":815104,"data_stored":333434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.42099999999999999,"15min":0.44},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.67600000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.42899999999999999,"15min":0.44500000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.54800000000000004}]},{"osd":1,"last update":"Sat Feb 10 05:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.45200000000000001,"15min":0.46999999999999997},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.58699999999999997,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.45600000000000002,"15min":0.47199999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.67000000000000004,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.501}]},{"osd":2,"last update":"Sat Feb 10 05:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.46200000000000002,"15min":0.47199999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.45200000000000001,"15min":0.46800000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.64500000000000002,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.41999999999999998}]},{"osd":4,"last update":"Sat Feb 10 05:55:36 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.58699999999999997,"15min":0.58399999999999996},"min":{"1min":0.44700000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.628,"5min":0.621,"15min":0.62},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.71499999999999997}]},{"osd":5,"last update":"Sat Feb 10 05:55:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.61899999999999999,"15min":0.625},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.623,"15min":0.621},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.78100000000000003}]},{"osd":6,"last update":"Sat Feb 10 05:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.72599999999999998}]},{"osd":7,"last update":"Sat Feb 10 05:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.88700000000000001}]}]},{"osd":4,"up_from":30,"seq":128849018915,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27808,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713248,"statfs":{"total":95990841344,"available":95962365952,"internally_reserved":0,"allocated":806912,"data_stored":332287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:55:35 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.54300000000000004,"15min":0.55100000000000005},"min":{"1min":0.38600000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.59399999999999997,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.437,"5min":0.377,"15min":0.377},"max":{"1min":0.68200000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.621}]},{"osd":1,"last update":"Sat Feb 10 05:55:35 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52500000000000002,"15min":0.53100000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.59699999999999998,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.53900000000000003,"15min":0.54400000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.64100000000000001,"5min":0.749,"15min":0.749},"last":0.80400000000000005}]},{"osd":2,"last update":"Sat Feb 10 05:55:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56399999999999995,"15min":0.56699999999999995},"min":{"1min":0.39600000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.67000000000000004,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.52600000000000002,"15min":0.53100000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.623,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.58699999999999997}]},{"osd":3,"last update":"Sat Feb 10 05:55:35 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.40799999999999997,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.68600000000000005,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.70799999999999996,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.55900000000000005}]},{"osd":5,"last update":"Sat Feb 10 05:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.432,"15min":0.438},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.56799999999999995,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.47699999999999998,"15min":0.47499999999999998},"min":{"1min":0.36399999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.71099999999999997}]},{"osd":6,"last update":"Sat Feb 10 05:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.77100000000000002}]},{"osd":7,"last update":"Sat Feb 10 05:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.69599999999999995}]}]},{"osd":5,"up_from":38,"seq":163208757278,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27788,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713268,"statfs":{"total":95990841344,"available":95962386432,"internally_reserved":0,"allocated":786432,"data_stored":332502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.55900000000000005,"15min":0.56699999999999995},"min":{"1min":0.35999999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.69999999999999996,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.73999999999999999,"5min":0.875,"15min":0.875},"last":0.54300000000000004}]},{"osd":1,"last update":"Sat Feb 10 05:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.747,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.55200000000000005,"15min":0.56200000000000006},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.63400000000000001,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.49399999999999999}]},{"osd":2,"last update":"Sat Feb 10 05:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.59799999999999998,"15min":0.61299999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.64600000000000002,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.64100000000000001,"15min":0.65600000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.82799999999999996,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":0.629}]},{"osd":3,"last update":"Sat Feb 10 05:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.44400000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.80900000000000005,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.58399999999999996,"15min":0.58999999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.67500000000000004,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.57099999999999995}]},{"osd":4,"last update":"Sat Feb 10 05:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39100000000000001,"15min":0.39000000000000001},"min":{"1min":0.30099999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.59799999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.38900000000000001,"15min":0.38500000000000001},"min":{"1min":0.28499999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.54800000000000004,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.47899999999999998}]},{"osd":6,"last update":"Sat Feb 10 05:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.42999999999999999}]},{"osd":7,"last update":"Sat Feb 10 05:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.374}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T05:56:14.232 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T05:56:14.484 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:56:14.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:14 smithi064 ceph-mon[25185]: Updating smithi176:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:56:14.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:14 smithi064 ceph-mon[25185]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:56:14.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:14 smithi064 ceph-mon[25185]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:56:14.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:14 smithi064 ceph-mon[25185]: Updating smithi064:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:56:14.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:14 smithi064 ceph-mon[25185]: Updating smithi176:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:56:14.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:14 smithi064 ceph-mon[25185]: Deploying daemon alertmanager.a on smithi064 2024-02-10T05:56:14.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:14 smithi064 ceph-mon[25185]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:56:14.595 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:14 smithi064 ceph-mon[38958]: Updating smithi176:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:56:14.595 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:14 smithi064 ceph-mon[38958]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:56:14.595 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:14 smithi064 ceph-mon[38958]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:56:14.595 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:14 smithi064 ceph-mon[38958]: Updating smithi064:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:56:14.595 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:14 smithi064 ceph-mon[38958]: Updating smithi176:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:56:14.595 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:14 smithi064 ceph-mon[38958]: Deploying daemon alertmanager.a on smithi064 2024-02-10T05:56:14.595 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:14 smithi064 ceph-mon[38958]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:56:14.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:14 smithi176 ceph-mon[30294]: Updating smithi176:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:56:14.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:14 smithi176 ceph-mon[30294]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:56:14.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:14 smithi176 ceph-mon[30294]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:56:14.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:14 smithi176 ceph-mon[30294]: Updating smithi064:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:56:14.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:14 smithi176 ceph-mon[30294]: Updating smithi176:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:56:14.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:14 smithi176 ceph-mon[30294]: Deploying daemon alertmanager.a on smithi064 2024-02-10T05:56:14.739 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:14 smithi176 ceph-mon[30294]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:56:15.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:15 smithi064 ceph-mon[25185]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:15.626 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:15 smithi064 ceph-mon[38958]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:15.738 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:15 smithi176 ceph-mon[30294]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:16.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:16 smithi064 ceph-mon[25185]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:16.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:16 smithi064 ceph-mon[38958]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:16.920 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:56:16.923 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2024-02-10T05:56:16.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:16 smithi176 ceph-mon[30294]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:17.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:17 smithi064 ceph-mon[25185]: from='client.24599 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:56:17.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:17 smithi176 ceph-mon[30294]: from='client.24599 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:56:18.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:17 smithi064 ceph-mon[38958]: from='client.24599 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:56:18.651 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-02-10T05:56:15.786576+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1320,"num_read_kb":1131,"num_write":873,"num_write_kb":4027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1771654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":854,"ondisk_log_size":854,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749928448,"kb_used":240784,"kb_used_data":7996,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749687664,"statfs":{"total":767926730752,"available":767680167936,"internally_reserved":0,"allocated":8187904,"data_stored":4433876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003061"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735275+0000","last_change":"2024-02-10T05:55:02.583157+0000","last_active":"2024-02-10T05:55:35.735275+0000","last_peered":"2024-02-10T05:55:35.735275+0000","last_clean":"2024-02-10T05:55:35.735275+0000","last_became_active":"2024-02-10T05:55:02.582860+0000","last_became_peered":"2024-02-10T05:55:02.582860+0000","last_unstale":"2024-02-10T05:55:35.735275+0000","last_undegraded":"2024-02-10T05:55:35.735275+0000","last_fullsized":"2024-02-10T05:55:35.735275+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:40:47.915444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.745577+0000","last_change":"2024-02-10T05:55:00.573270+0000","last_active":"2024-02-10T05:55:35.745577+0000","last_peered":"2024-02-10T05:55:35.745577+0000","last_clean":"2024-02-10T05:55:35.745577+0000","last_became_active":"2024-02-10T05:55:00.572941+0000","last_became_peered":"2024-02-10T05:55:00.572941+0000","last_unstale":"2024-02-10T05:55:35.745577+0000","last_undegraded":"2024-02-10T05:55:35.745577+0000","last_fullsized":"2024-02-10T05:55:35.745577+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:47:13.730741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.740323+0000","last_change":"2024-02-10T05:54:56.533959+0000","last_active":"2024-02-10T05:55:35.740323+0000","last_peered":"2024-02-10T05:55:35.740323+0000","last_clean":"2024-02-10T05:55:35.740323+0000","last_became_active":"2024-02-10T05:54:56.533570+0000","last_became_peered":"2024-02-10T05:54:56.533570+0000","last_unstale":"2024-02-10T05:55:35.740323+0000","last_undegraded":"2024-02-10T05:55:35.740323+0000","last_fullsized":"2024-02-10T05:55:35.740323+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:56:22.136122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'15","reported_seq":75,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735029+0000","last_change":"2024-02-10T05:55:25.937510+0000","last_active":"2024-02-10T05:55:35.735029+0000","last_peered":"2024-02-10T05:55:35.735029+0000","last_clean":"2024-02-10T05:55:35.735029+0000","last_became_active":"2024-02-10T05:55:25.924844+0000","last_became_peered":"2024-02-10T05:55:25.924844+0000","last_unstale":"2024-02-10T05:55:35.735029+0000","last_undegraded":"2024-02-10T05:55:35.735029+0000","last_fullsized":"2024-02-10T05:55:35.735029+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:44:00.040076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338286+0000","last_change":"2024-02-10T05:55:03.129505+0000","last_active":"2024-02-10T05:55:36.338286+0000","last_peered":"2024-02-10T05:55:36.338286+0000","last_clean":"2024-02-10T05:55:36.338286+0000","last_became_active":"2024-02-10T05:55:03.129113+0000","last_became_peered":"2024-02-10T05:55:03.129113+0000","last_unstale":"2024-02-10T05:55:36.338286+0000","last_undegraded":"2024-02-10T05:55:36.338286+0000","last_fullsized":"2024-02-10T05:55:36.338286+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:54:52.858555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735034+0000","last_change":"2024-02-10T05:55:00.573765+0000","last_active":"2024-02-10T05:55:35.735034+0000","last_peered":"2024-02-10T05:55:35.735034+0000","last_clean":"2024-02-10T05:55:35.735034+0000","last_became_active":"2024-02-10T05:55:00.573432+0000","last_became_peered":"2024-02-10T05:55:00.573432+0000","last_unstale":"2024-02-10T05:55:35.735034+0000","last_undegraded":"2024-02-10T05:55:35.735034+0000","last_fullsized":"2024-02-10T05:55:35.735034+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:39:15.660426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.339037+0000","last_change":"2024-02-10T05:54:56.543724+0000","last_active":"2024-02-10T05:55:36.339037+0000","last_peered":"2024-02-10T05:55:36.339037+0000","last_clean":"2024-02-10T05:55:36.339037+0000","last_became_active":"2024-02-10T05:54:56.543445+0000","last_became_peered":"2024-02-10T05:54:56.543445+0000","last_unstale":"2024-02-10T05:55:36.339037+0000","last_undegraded":"2024-02-10T05:55:36.339037+0000","last_fullsized":"2024-02-10T05:55:36.339037+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:07:42.871971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'39","reported_seq":102,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338199+0000","last_change":"2024-02-10T05:54:58.565969+0000","last_active":"2024-02-10T05:55:36.338199+0000","last_peered":"2024-02-10T05:55:36.338199+0000","last_clean":"2024-02-10T05:55:36.338199+0000","last_became_active":"2024-02-10T05:54:58.565454+0000","last_became_peered":"2024-02-10T05:54:58.565454+0000","last_unstale":"2024-02-10T05:55:36.338199+0000","last_undegraded":"2024-02-10T05:55:36.338199+0000","last_fullsized":"2024-02-10T05:55:36.338199+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:08:42.374580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.740792+0000","last_change":"2024-02-10T05:55:02.588648+0000","last_active":"2024-02-10T05:55:35.740792+0000","last_peered":"2024-02-10T05:55:35.740792+0000","last_clean":"2024-02-10T05:55:35.740792+0000","last_became_active":"2024-02-10T05:55:02.588008+0000","last_became_peered":"2024-02-10T05:55:02.588008+0000","last_unstale":"2024-02-10T05:55:35.740792+0000","last_undegraded":"2024-02-10T05:55:35.740792+0000","last_fullsized":"2024-02-10T05:55:35.740792+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:27:23.404755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014343+0000","last_change":"2024-02-10T05:55:00.571339+0000","last_active":"2024-02-10T05:55:36.014343+0000","last_peered":"2024-02-10T05:55:36.014343+0000","last_clean":"2024-02-10T05:55:36.014343+0000","last_became_active":"2024-02-10T05:55:00.571062+0000","last_became_peered":"2024-02-10T05:55:00.571062+0000","last_unstale":"2024-02-10T05:55:36.014343+0000","last_undegraded":"2024-02-10T05:55:36.014343+0000","last_fullsized":"2024-02-10T05:55:36.014343+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:44:05.542585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014613+0000","last_change":"2024-02-10T05:54:56.537716+0000","last_active":"2024-02-10T05:55:36.014613+0000","last_peered":"2024-02-10T05:55:36.014613+0000","last_clean":"2024-02-10T05:55:36.014613+0000","last_became_active":"2024-02-10T05:54:56.537413+0000","last_became_peered":"2024-02-10T05:54:56.537413+0000","last_unstale":"2024-02-10T05:55:36.014613+0000","last_undegraded":"2024-02-10T05:55:36.014613+0000","last_fullsized":"2024-02-10T05:55:36.014613+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:24:13.401436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'21","reported_seq":69,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735478+0000","last_change":"2024-02-10T05:54:58.565422+0000","last_active":"2024-02-10T05:55:35.735478+0000","last_peered":"2024-02-10T05:55:35.735478+0000","last_clean":"2024-02-10T05:55:35.735478+0000","last_became_active":"2024-02-10T05:54:58.564397+0000","last_became_peered":"2024-02-10T05:54:58.564397+0000","last_unstale":"2024-02-10T05:55:35.735478+0000","last_undegraded":"2024-02-10T05:55:35.735478+0000","last_fullsized":"2024-02-10T05:55:35.735478+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:28:10.134384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014213+0000","last_change":"2024-02-10T05:55:02.584058+0000","last_active":"2024-02-10T05:55:36.014213+0000","last_peered":"2024-02-10T05:55:36.014213+0000","last_clean":"2024-02-10T05:55:36.014213+0000","last_became_active":"2024-02-10T05:55:02.583691+0000","last_became_peered":"2024-02-10T05:55:02.583691+0000","last_unstale":"2024-02-10T05:55:36.014213+0000","last_undegraded":"2024-02-10T05:55:36.014213+0000","last_fullsized":"2024-02-10T05:55:36.014213+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:58:13.800252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.030938+0000","last_change":"2024-02-10T05:55:00.570365+0000","last_active":"2024-02-10T05:55:36.030938+0000","last_peered":"2024-02-10T05:55:36.030938+0000","last_clean":"2024-02-10T05:55:36.030938+0000","last_became_active":"2024-02-10T05:55:00.569976+0000","last_became_peered":"2024-02-10T05:55:00.569976+0000","last_unstale":"2024-02-10T05:55:36.030938+0000","last_undegraded":"2024-02-10T05:55:36.030938+0000","last_fullsized":"2024-02-10T05:55:36.030938+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:39:40.031844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014556+0000","last_change":"2024-02-10T05:54:56.539557+0000","last_active":"2024-02-10T05:55:36.014556+0000","last_peered":"2024-02-10T05:55:36.014556+0000","last_clean":"2024-02-10T05:55:36.014556+0000","last_became_active":"2024-02-10T05:54:56.539244+0000","last_became_peered":"2024-02-10T05:54:56.539244+0000","last_unstale":"2024-02-10T05:55:36.014556+0000","last_undegraded":"2024-02-10T05:55:36.014556+0000","last_fullsized":"2024-02-10T05:55:36.014556+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:06:03.603419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'11","reported_seq":51,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735320+0000","last_change":"2024-02-10T05:54:58.566132+0000","last_active":"2024-02-10T05:55:35.735320+0000","last_peered":"2024-02-10T05:55:35.735320+0000","last_clean":"2024-02-10T05:55:35.735320+0000","last_became_active":"2024-02-10T05:54:58.565645+0000","last_became_peered":"2024-02-10T05:54:58.565645+0000","last_unstale":"2024-02-10T05:55:35.735320+0000","last_undegraded":"2024-02-10T05:55:35.735320+0000","last_fullsized":"2024-02-10T05:55:35.735320+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:40:12.776793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735757+0000","last_change":"2024-02-10T05:55:03.131517+0000","last_active":"2024-02-10T05:55:35.735757+0000","last_peered":"2024-02-10T05:55:35.735757+0000","last_clean":"2024-02-10T05:55:35.735757+0000","last_became_active":"2024-02-10T05:55:03.129312+0000","last_became_peered":"2024-02-10T05:55:03.129312+0000","last_unstale":"2024-02-10T05:55:35.735757+0000","last_undegraded":"2024-02-10T05:55:35.735757+0000","last_fullsized":"2024-02-10T05:55:35.735757+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:32:26.382992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.745162+0000","last_change":"2024-02-10T05:55:00.568070+0000","last_active":"2024-02-10T05:55:35.745162+0000","last_peered":"2024-02-10T05:55:35.745162+0000","last_clean":"2024-02-10T05:55:35.745162+0000","last_became_active":"2024-02-10T05:55:00.567688+0000","last_became_peered":"2024-02-10T05:55:00.567688+0000","last_unstale":"2024-02-10T05:55:35.745162+0000","last_undegraded":"2024-02-10T05:55:35.745162+0000","last_fullsized":"2024-02-10T05:55:35.745162+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:11:25.503197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.744260+0000","last_change":"2024-02-10T05:54:56.537339+0000","last_active":"2024-02-10T05:55:35.744260+0000","last_peered":"2024-02-10T05:55:35.744260+0000","last_clean":"2024-02-10T05:55:35.744260+0000","last_became_active":"2024-02-10T05:54:56.537037+0000","last_became_peered":"2024-02-10T05:54:56.537037+0000","last_unstale":"2024-02-10T05:55:35.744260+0000","last_undegraded":"2024-02-10T05:55:35.744260+0000","last_fullsized":"2024-02-10T05:55:35.744260+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:19:54.965360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.027415+0000","last_change":"2024-02-10T05:54:58.560807+0000","last_active":"2024-02-10T05:55:36.027415+0000","last_peered":"2024-02-10T05:55:36.027415+0000","last_clean":"2024-02-10T05:55:36.027415+0000","last_became_active":"2024-02-10T05:54:58.560477+0000","last_became_peered":"2024-02-10T05:54:58.560477+0000","last_unstale":"2024-02-10T05:55:36.027415+0000","last_undegraded":"2024-02-10T05:55:36.027415+0000","last_fullsized":"2024-02-10T05:55:36.027415+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:26:46.496770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.339115+0000","last_change":"2024-02-10T05:55:03.129750+0000","last_active":"2024-02-10T05:55:36.339115+0000","last_peered":"2024-02-10T05:55:36.339115+0000","last_clean":"2024-02-10T05:55:36.339115+0000","last_became_active":"2024-02-10T05:55:03.129430+0000","last_became_peered":"2024-02-10T05:55:03.129430+0000","last_unstale":"2024-02-10T05:55:36.339115+0000","last_undegraded":"2024-02-10T05:55:36.339115+0000","last_fullsized":"2024-02-10T05:55:36.339115+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:22:50.301437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735954+0000","last_change":"2024-02-10T05:55:00.575265+0000","last_active":"2024-02-10T05:55:35.735954+0000","last_peered":"2024-02-10T05:55:35.735954+0000","last_clean":"2024-02-10T05:55:35.735954+0000","last_became_active":"2024-02-10T05:55:00.574722+0000","last_became_peered":"2024-02-10T05:55:00.574722+0000","last_unstale":"2024-02-10T05:55:35.735954+0000","last_undegraded":"2024-02-10T05:55:35.735954+0000","last_fullsized":"2024-02-10T05:55:35.735954+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:36:44.376095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735605+0000","last_change":"2024-02-10T05:54:56.541416+0000","last_active":"2024-02-10T05:55:35.735605+0000","last_peered":"2024-02-10T05:55:35.735605+0000","last_clean":"2024-02-10T05:55:35.735605+0000","last_became_active":"2024-02-10T05:54:56.541127+0000","last_became_peered":"2024-02-10T05:54:56.541127+0000","last_unstale":"2024-02-10T05:55:35.735605+0000","last_undegraded":"2024-02-10T05:55:35.735605+0000","last_fullsized":"2024-02-10T05:55:35.735605+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:06:08.251992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'18","reported_seq":64,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338797+0000","last_change":"2024-02-10T05:54:58.566311+0000","last_active":"2024-02-10T05:55:36.338797+0000","last_peered":"2024-02-10T05:55:36.338797+0000","last_clean":"2024-02-10T05:55:36.338797+0000","last_became_active":"2024-02-10T05:54:58.566016+0000","last_became_peered":"2024-02-10T05:54:58.566016+0000","last_unstale":"2024-02-10T05:55:36.338797+0000","last_undegraded":"2024-02-10T05:55:36.338797+0000","last_fullsized":"2024-02-10T05:55:36.338797+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:34:43.295518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.031259+0000","last_change":"2024-02-10T05:55:02.580987+0000","last_active":"2024-02-10T05:55:36.031259+0000","last_peered":"2024-02-10T05:55:36.031259+0000","last_clean":"2024-02-10T05:55:36.031259+0000","last_became_active":"2024-02-10T05:55:02.580621+0000","last_became_peered":"2024-02-10T05:55:02.580621+0000","last_unstale":"2024-02-10T05:55:36.031259+0000","last_undegraded":"2024-02-10T05:55:36.031259+0000","last_fullsized":"2024-02-10T05:55:36.031259+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:07:00.603547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"67'8","reported_seq":52,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:56:06.597577+0000","last_change":"2024-02-10T05:55:00.573585+0000","last_active":"2024-02-10T05:56:06.597577+0000","last_peered":"2024-02-10T05:56:06.597577+0000","last_clean":"2024-02-10T05:56:06.597577+0000","last_became_active":"2024-02-10T05:55:00.573247+0000","last_became_peered":"2024-02-10T05:55:00.573247+0000","last_unstale":"2024-02-10T05:56:06.597577+0000","last_undegraded":"2024-02-10T05:56:06.597577+0000","last_fullsized":"2024-02-10T05:56:06.597577+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:39:52.075233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.745659+0000","last_change":"2024-02-10T05:54:56.545093+0000","last_active":"2024-02-10T05:55:35.745659+0000","last_peered":"2024-02-10T05:55:35.745659+0000","last_clean":"2024-02-10T05:55:35.745659+0000","last_became_active":"2024-02-10T05:54:56.544798+0000","last_became_peered":"2024-02-10T05:54:56.544798+0000","last_unstale":"2024-02-10T05:55:35.745659+0000","last_undegraded":"2024-02-10T05:55:35.745659+0000","last_fullsized":"2024-02-10T05:55:35.745659+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:53:01.776249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'16","reported_seq":56,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.741760+0000","last_change":"2024-02-10T05:54:58.556890+0000","last_active":"2024-02-10T05:55:35.741760+0000","last_peered":"2024-02-10T05:55:35.741760+0000","last_clean":"2024-02-10T05:55:35.741760+0000","last_became_active":"2024-02-10T05:54:58.555915+0000","last_became_peered":"2024-02-10T05:54:58.555915+0000","last_unstale":"2024-02-10T05:55:35.741760+0000","last_undegraded":"2024-02-10T05:55:35.741760+0000","last_fullsized":"2024-02-10T05:55:35.741760+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:46:57.139563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.745323+0000","last_change":"2024-02-10T05:55:02.586369+0000","last_active":"2024-02-10T05:55:35.745323+0000","last_peered":"2024-02-10T05:55:35.745323+0000","last_clean":"2024-02-10T05:55:35.745323+0000","last_became_active":"2024-02-10T05:55:02.585359+0000","last_became_peered":"2024-02-10T05:55:02.585359+0000","last_unstale":"2024-02-10T05:55:35.745323+0000","last_undegraded":"2024-02-10T05:55:35.745323+0000","last_fullsized":"2024-02-10T05:55:35.745323+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:55:53.925628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.742163+0000","last_change":"2024-02-10T05:55:00.567096+0000","last_active":"2024-02-10T05:55:35.742163+0000","last_peered":"2024-02-10T05:55:35.742163+0000","last_clean":"2024-02-10T05:55:35.742163+0000","last_became_active":"2024-02-10T05:55:00.566767+0000","last_became_peered":"2024-02-10T05:55:00.566767+0000","last_unstale":"2024-02-10T05:55:35.742163+0000","last_undegraded":"2024-02-10T05:55:35.742163+0000","last_fullsized":"2024-02-10T05:55:35.742163+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:29:13.795369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.337945+0000","last_change":"2024-02-10T05:54:56.544008+0000","last_active":"2024-02-10T05:55:36.337945+0000","last_peered":"2024-02-10T05:55:36.337945+0000","last_clean":"2024-02-10T05:55:36.337945+0000","last_became_active":"2024-02-10T05:54:56.543695+0000","last_became_peered":"2024-02-10T05:54:56.543695+0000","last_unstale":"2024-02-10T05:55:36.337945+0000","last_undegraded":"2024-02-10T05:55:36.337945+0000","last_fullsized":"2024-02-10T05:55:36.337945+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:27:24.021139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'17","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338576+0000","last_change":"2024-02-10T05:54:58.559748+0000","last_active":"2024-02-10T05:55:36.338576+0000","last_peered":"2024-02-10T05:55:36.338576+0000","last_clean":"2024-02-10T05:55:36.338576+0000","last_became_active":"2024-02-10T05:54:58.559149+0000","last_became_peered":"2024-02-10T05:54:58.559149+0000","last_unstale":"2024-02-10T05:55:36.338576+0000","last_undegraded":"2024-02-10T05:55:36.338576+0000","last_fullsized":"2024-02-10T05:55:36.338576+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:55:47.488264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"67'1","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.030936+0000","last_change":"2024-02-10T05:55:02.581397+0000","last_active":"2024-02-10T05:55:36.030936+0000","last_peered":"2024-02-10T05:55:36.030936+0000","last_clean":"2024-02-10T05:55:36.030936+0000","last_became_active":"2024-02-10T05:55:02.580933+0000","last_became_peered":"2024-02-10T05:55:02.580933+0000","last_unstale":"2024-02-10T05:55:36.030936+0000","last_undegraded":"2024-02-10T05:55:36.030936+0000","last_fullsized":"2024-02-10T05:55:36.030936+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:43:14.953573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338025+0000","last_change":"2024-02-10T05:55:00.575266+0000","last_active":"2024-02-10T05:55:36.338025+0000","last_peered":"2024-02-10T05:55:36.338025+0000","last_clean":"2024-02-10T05:55:36.338025+0000","last_became_active":"2024-02-10T05:55:00.574983+0000","last_became_peered":"2024-02-10T05:55:00.574983+0000","last_unstale":"2024-02-10T05:55:36.338025+0000","last_undegraded":"2024-02-10T05:55:36.338025+0000","last_fullsized":"2024-02-10T05:55:36.338025+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:02:01.862287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.740406+0000","last_change":"2024-02-10T05:54:56.545934+0000","last_active":"2024-02-10T05:55:35.740406+0000","last_peered":"2024-02-10T05:55:35.740406+0000","last_clean":"2024-02-10T05:55:35.740406+0000","last_became_active":"2024-02-10T05:54:56.545572+0000","last_became_peered":"2024-02-10T05:54:56.545572+0000","last_unstale":"2024-02-10T05:55:35.740406+0000","last_undegraded":"2024-02-10T05:55:35.740406+0000","last_fullsized":"2024-02-10T05:55:35.740406+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:00:41.770228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":35,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.337992+0000","last_change":"2024-02-10T05:54:58.562130+0000","last_active":"2024-02-10T05:55:36.337992+0000","last_peered":"2024-02-10T05:55:36.337992+0000","last_clean":"2024-02-10T05:55:36.337992+0000","last_became_active":"2024-02-10T05:54:58.561805+0000","last_became_peered":"2024-02-10T05:54:58.561805+0000","last_unstale":"2024-02-10T05:55:36.337992+0000","last_undegraded":"2024-02-10T05:55:36.337992+0000","last_fullsized":"2024-02-10T05:55:36.337992+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:58:07.278752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338146+0000","last_change":"2024-02-10T05:55:03.129793+0000","last_active":"2024-02-10T05:55:36.338146+0000","last_peered":"2024-02-10T05:55:36.338146+0000","last_clean":"2024-02-10T05:55:36.338146+0000","last_became_active":"2024-02-10T05:55:03.129134+0000","last_became_peered":"2024-02-10T05:55:03.129134+0000","last_unstale":"2024-02-10T05:55:36.338146+0000","last_undegraded":"2024-02-10T05:55:36.338146+0000","last_fullsized":"2024-02-10T05:55:36.338146+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:11:59.629570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.031137+0000","last_change":"2024-02-10T05:55:00.573837+0000","last_active":"2024-02-10T05:55:36.031137+0000","last_peered":"2024-02-10T05:55:36.031137+0000","last_clean":"2024-02-10T05:55:36.031137+0000","last_became_active":"2024-02-10T05:55:00.573556+0000","last_became_peered":"2024-02-10T05:55:00.573556+0000","last_unstale":"2024-02-10T05:55:36.031137+0000","last_undegraded":"2024-02-10T05:55:36.031137+0000","last_fullsized":"2024-02-10T05:55:36.031137+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:01:52.648894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014806+0000","last_change":"2024-02-10T05:54:56.540211+0000","last_active":"2024-02-10T05:55:36.014806+0000","last_peered":"2024-02-10T05:55:36.014806+0000","last_clean":"2024-02-10T05:55:36.014806+0000","last_became_active":"2024-02-10T05:54:56.539924+0000","last_became_peered":"2024-02-10T05:54:56.539924+0000","last_unstale":"2024-02-10T05:55:36.014806+0000","last_undegraded":"2024-02-10T05:55:36.014806+0000","last_fullsized":"2024-02-10T05:55:36.014806+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:20:49.267434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"67'17","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338179+0000","last_change":"2024-02-10T05:54:58.562915+0000","last_active":"2024-02-10T05:55:36.338179+0000","last_peered":"2024-02-10T05:55:36.338179+0000","last_clean":"2024-02-10T05:55:36.338179+0000","last_became_active":"2024-02-10T05:54:58.562614+0000","last_became_peered":"2024-02-10T05:54:58.562614+0000","last_unstale":"2024-02-10T05:55:36.338179+0000","last_undegraded":"2024-02-10T05:55:36.338179+0000","last_fullsized":"2024-02-10T05:55:36.338179+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:14:19.106756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.738481+0000","last_change":"2024-02-10T05:55:02.588905+0000","last_active":"2024-02-10T05:55:35.738481+0000","last_peered":"2024-02-10T05:55:35.738481+0000","last_clean":"2024-02-10T05:55:35.738481+0000","last_became_active":"2024-02-10T05:55:02.588444+0000","last_became_peered":"2024-02-10T05:55:02.588444+0000","last_unstale":"2024-02-10T05:55:35.738481+0000","last_undegraded":"2024-02-10T05:55:35.738481+0000","last_fullsized":"2024-02-10T05:55:35.738481+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:47:18.063995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338402+0000","last_change":"2024-02-10T05:55:00.579149+0000","last_active":"2024-02-10T05:55:36.338402+0000","last_peered":"2024-02-10T05:55:36.338402+0000","last_clean":"2024-02-10T05:55:36.338402+0000","last_became_active":"2024-02-10T05:55:00.578524+0000","last_became_peered":"2024-02-10T05:55:00.578524+0000","last_unstale":"2024-02-10T05:55:36.338402+0000","last_undegraded":"2024-02-10T05:55:36.338402+0000","last_fullsized":"2024-02-10T05:55:36.338402+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:57:12.009589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.031335+0000","last_change":"2024-02-10T05:54:56.544187+0000","last_active":"2024-02-10T05:55:36.031335+0000","last_peered":"2024-02-10T05:55:36.031335+0000","last_clean":"2024-02-10T05:55:36.031335+0000","last_became_active":"2024-02-10T05:54:56.543884+0000","last_became_peered":"2024-02-10T05:54:56.543884+0000","last_unstale":"2024-02-10T05:55:36.031335+0000","last_undegraded":"2024-02-10T05:55:36.031335+0000","last_fullsized":"2024-02-10T05:55:36.031335+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:39:54.458211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.743270+0000","last_change":"2024-02-10T05:54:58.555538+0000","last_active":"2024-02-10T05:55:35.743270+0000","last_peered":"2024-02-10T05:55:35.743270+0000","last_clean":"2024-02-10T05:55:35.743270+0000","last_became_active":"2024-02-10T05:54:58.555218+0000","last_became_peered":"2024-02-10T05:54:58.555218+0000","last_unstale":"2024-02-10T05:55:35.743270+0000","last_undegraded":"2024-02-10T05:55:35.743270+0000","last_fullsized":"2024-02-10T05:55:35.743270+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:58:13.440955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.337967+0000","last_change":"2024-02-10T05:55:02.587907+0000","last_active":"2024-02-10T05:55:36.337967+0000","last_peered":"2024-02-10T05:55:36.337967+0000","last_clean":"2024-02-10T05:55:36.337967+0000","last_became_active":"2024-02-10T05:55:02.587611+0000","last_became_peered":"2024-02-10T05:55:02.587611+0000","last_unstale":"2024-02-10T05:55:36.337967+0000","last_undegraded":"2024-02-10T05:55:36.337967+0000","last_fullsized":"2024-02-10T05:55:36.337967+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:31:30.435093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.743056+0000","last_change":"2024-02-10T05:55:00.577369+0000","last_active":"2024-02-10T05:55:35.743056+0000","last_peered":"2024-02-10T05:55:35.743056+0000","last_clean":"2024-02-10T05:55:35.743056+0000","last_became_active":"2024-02-10T05:55:00.577073+0000","last_became_peered":"2024-02-10T05:55:00.577073+0000","last_unstale":"2024-02-10T05:55:35.743056+0000","last_undegraded":"2024-02-10T05:55:35.743056+0000","last_fullsized":"2024-02-10T05:55:35.743056+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:00:47.714540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.736227+0000","last_change":"2024-02-10T05:54:56.535861+0000","last_active":"2024-02-10T05:55:35.736227+0000","last_peered":"2024-02-10T05:55:35.736227+0000","last_clean":"2024-02-10T05:55:35.736227+0000","last_became_active":"2024-02-10T05:54:56.535593+0000","last_became_peered":"2024-02-10T05:54:56.535593+0000","last_unstale":"2024-02-10T05:55:35.736227+0000","last_undegraded":"2024-02-10T05:55:35.736227+0000","last_fullsized":"2024-02-10T05:55:35.736227+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:39:33.156246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735533+0000","last_change":"2024-02-10T05:54:58.563664+0000","last_active":"2024-02-10T05:55:35.735533+0000","last_peered":"2024-02-10T05:55:35.735533+0000","last_clean":"2024-02-10T05:55:35.735533+0000","last_became_active":"2024-02-10T05:54:58.562365+0000","last_became_peered":"2024-02-10T05:54:58.562365+0000","last_unstale":"2024-02-10T05:55:35.735533+0000","last_undegraded":"2024-02-10T05:55:35.735533+0000","last_fullsized":"2024-02-10T05:55:35.735533+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:17:29.418689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.741042+0000","last_change":"2024-02-10T05:55:02.585145+0000","last_active":"2024-02-10T05:55:35.741042+0000","last_peered":"2024-02-10T05:55:35.741042+0000","last_clean":"2024-02-10T05:55:35.741042+0000","last_became_active":"2024-02-10T05:55:02.584812+0000","last_became_peered":"2024-02-10T05:55:02.584812+0000","last_unstale":"2024-02-10T05:55:35.741042+0000","last_undegraded":"2024-02-10T05:55:35.741042+0000","last_fullsized":"2024-02-10T05:55:35.741042+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:31:21.797773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:56:06.597580+0000","last_change":"2024-02-10T05:55:00.572045+0000","last_active":"2024-02-10T05:56:06.597580+0000","last_peered":"2024-02-10T05:56:06.597580+0000","last_clean":"2024-02-10T05:56:06.597580+0000","last_became_active":"2024-02-10T05:55:00.571758+0000","last_became_peered":"2024-02-10T05:55:00.571758+0000","last_unstale":"2024-02-10T05:56:06.597580+0000","last_undegraded":"2024-02-10T05:56:06.597580+0000","last_fullsized":"2024-02-10T05:56:06.597580+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:57:10.810245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.031196+0000","last_change":"2024-02-10T05:54:56.542343+0000","last_active":"2024-02-10T05:55:36.031196+0000","last_peered":"2024-02-10T05:55:36.031196+0000","last_clean":"2024-02-10T05:55:36.031196+0000","last_became_active":"2024-02-10T05:54:56.541409+0000","last_became_peered":"2024-02-10T05:54:56.541409+0000","last_unstale":"2024-02-10T05:55:36.031196+0000","last_undegraded":"2024-02-10T05:55:36.031196+0000","last_fullsized":"2024-02-10T05:55:36.031196+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:45:17.512782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"67'22","reported_seq":68,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338553+0000","last_change":"2024-02-10T05:54:58.561627+0000","last_active":"2024-02-10T05:55:36.338553+0000","last_peered":"2024-02-10T05:55:36.338553+0000","last_clean":"2024-02-10T05:55:36.338553+0000","last_became_active":"2024-02-10T05:54:58.560947+0000","last_became_peered":"2024-02-10T05:54:58.560947+0000","last_unstale":"2024-02-10T05:55:36.338553+0000","last_undegraded":"2024-02-10T05:55:36.338553+0000","last_fullsized":"2024-02-10T05:55:36.338553+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:03:16.214090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.736472+0000","last_change":"2024-02-10T05:55:02.581266+0000","last_active":"2024-02-10T05:55:35.736472+0000","last_peered":"2024-02-10T05:55:35.736472+0000","last_clean":"2024-02-10T05:55:35.736472+0000","last_became_active":"2024-02-10T05:55:02.580728+0000","last_became_peered":"2024-02-10T05:55:02.580728+0000","last_unstale":"2024-02-10T05:55:35.736472+0000","last_undegraded":"2024-02-10T05:55:35.736472+0000","last_fullsized":"2024-02-10T05:55:35.736472+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:22:22.437287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:56:06.597437+0000","last_change":"2024-02-10T05:55:00.572209+0000","last_active":"2024-02-10T05:56:06.597437+0000","last_peered":"2024-02-10T05:56:06.597437+0000","last_clean":"2024-02-10T05:56:06.597437+0000","last_became_active":"2024-02-10T05:55:00.571781+0000","last_became_peered":"2024-02-10T05:55:00.571781+0000","last_unstale":"2024-02-10T05:56:06.597437+0000","last_undegraded":"2024-02-10T05:56:06.597437+0000","last_fullsized":"2024-02-10T05:56:06.597437+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:02:56.718822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.739527+0000","last_change":"2024-02-10T05:54:56.543080+0000","last_active":"2024-02-10T05:55:35.739527+0000","last_peered":"2024-02-10T05:55:35.739527+0000","last_clean":"2024-02-10T05:55:35.739527+0000","last_became_active":"2024-02-10T05:54:56.542777+0000","last_became_peered":"2024-02-10T05:54:56.542777+0000","last_unstale":"2024-02-10T05:55:35.739527+0000","last_undegraded":"2024-02-10T05:55:35.739527+0000","last_fullsized":"2024-02-10T05:55:35.739527+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:38:59.251107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'15","reported_seq":57,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014876+0000","last_change":"2024-02-10T05:54:58.557620+0000","last_active":"2024-02-10T05:55:36.014876+0000","last_peered":"2024-02-10T05:55:36.014876+0000","last_clean":"2024-02-10T05:55:36.014876+0000","last_became_active":"2024-02-10T05:54:58.557283+0000","last_became_peered":"2024-02-10T05:54:58.557283+0000","last_unstale":"2024-02-10T05:55:36.014876+0000","last_undegraded":"2024-02-10T05:55:36.014876+0000","last_fullsized":"2024-02-10T05:55:36.014876+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:33:21.965789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.027640+0000","last_change":"2024-02-10T05:55:02.579787+0000","last_active":"2024-02-10T05:55:36.027640+0000","last_peered":"2024-02-10T05:55:36.027640+0000","last_clean":"2024-02-10T05:55:36.027640+0000","last_became_active":"2024-02-10T05:55:02.579393+0000","last_became_peered":"2024-02-10T05:55:02.579393+0000","last_unstale":"2024-02-10T05:55:36.027640+0000","last_undegraded":"2024-02-10T05:55:36.027640+0000","last_fullsized":"2024-02-10T05:55:36.027640+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:39:08.433187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.339117+0000","last_change":"2024-02-10T05:55:00.577404+0000","last_active":"2024-02-10T05:55:36.339117+0000","last_peered":"2024-02-10T05:55:36.339117+0000","last_clean":"2024-02-10T05:55:36.339117+0000","last_became_active":"2024-02-10T05:55:00.577117+0000","last_became_peered":"2024-02-10T05:55:00.577117+0000","last_unstale":"2024-02-10T05:55:36.339117+0000","last_undegraded":"2024-02-10T05:55:36.339117+0000","last_fullsized":"2024-02-10T05:55:36.339117+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:48:02.229013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.030865+0000","last_change":"2024-02-10T05:54:56.536012+0000","last_active":"2024-02-10T05:55:36.030865+0000","last_peered":"2024-02-10T05:55:36.030865+0000","last_clean":"2024-02-10T05:55:36.030865+0000","last_became_active":"2024-02-10T05:54:56.535738+0000","last_became_peered":"2024-02-10T05:54:56.535738+0000","last_unstale":"2024-02-10T05:55:36.030865+0000","last_undegraded":"2024-02-10T05:55:36.030865+0000","last_fullsized":"2024-02-10T05:55:36.030865+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:10:33.020334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'21","reported_seq":69,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.741937+0000","last_change":"2024-02-10T05:54:58.556634+0000","last_active":"2024-02-10T05:55:35.741937+0000","last_peered":"2024-02-10T05:55:35.741937+0000","last_clean":"2024-02-10T05:55:35.741937+0000","last_became_active":"2024-02-10T05:54:58.555438+0000","last_became_peered":"2024-02-10T05:54:58.555438+0000","last_unstale":"2024-02-10T05:55:35.741937+0000","last_undegraded":"2024-02-10T05:55:35.741937+0000","last_fullsized":"2024-02-10T05:55:35.741937+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:01:35.640408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.031308+0000","last_change":"2024-02-10T05:55:03.130669+0000","last_active":"2024-02-10T05:55:36.031308+0000","last_peered":"2024-02-10T05:55:36.031308+0000","last_clean":"2024-02-10T05:55:36.031308+0000","last_became_active":"2024-02-10T05:55:03.130187+0000","last_became_peered":"2024-02-10T05:55:03.130187+0000","last_unstale":"2024-02-10T05:55:36.031308+0000","last_undegraded":"2024-02-10T05:55:36.031308+0000","last_fullsized":"2024-02-10T05:55:36.031308+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:30:02.853983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014858+0000","last_change":"2024-02-10T05:55:00.572961+0000","last_active":"2024-02-10T05:55:36.014858+0000","last_peered":"2024-02-10T05:55:36.014858+0000","last_clean":"2024-02-10T05:55:36.014858+0000","last_became_active":"2024-02-10T05:55:00.572419+0000","last_became_peered":"2024-02-10T05:55:00.572419+0000","last_unstale":"2024-02-10T05:55:36.014858+0000","last_undegraded":"2024-02-10T05:55:36.014858+0000","last_fullsized":"2024-02-10T05:55:36.014858+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:52:42.978728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"60'1","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.337972+0000","last_change":"2024-02-10T05:54:56.538297+0000","last_active":"2024-02-10T05:55:36.337972+0000","last_peered":"2024-02-10T05:55:36.337972+0000","last_clean":"2024-02-10T05:55:36.337972+0000","last_became_active":"2024-02-10T05:54:56.538037+0000","last_became_peered":"2024-02-10T05:54:56.538037+0000","last_unstale":"2024-02-10T05:55:36.337972+0000","last_undegraded":"2024-02-10T05:55:36.337972+0000","last_fullsized":"2024-02-10T05:55:36.337972+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:34:40.947680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'12","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.339167+0000","last_change":"2024-02-10T05:54:58.561576+0000","last_active":"2024-02-10T05:55:36.339167+0000","last_peered":"2024-02-10T05:55:36.339167+0000","last_clean":"2024-02-10T05:55:36.339167+0000","last_became_active":"2024-02-10T05:54:58.560880+0000","last_became_peered":"2024-02-10T05:54:58.560880+0000","last_unstale":"2024-02-10T05:55:36.339167+0000","last_undegraded":"2024-02-10T05:55:36.339167+0000","last_fullsized":"2024-02-10T05:55:36.339167+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:16:11.344714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014511+0000","last_change":"2024-02-10T05:55:03.129351+0000","last_active":"2024-02-10T05:55:36.014511+0000","last_peered":"2024-02-10T05:55:36.014511+0000","last_clean":"2024-02-10T05:55:36.014511+0000","last_became_active":"2024-02-10T05:55:03.129054+0000","last_became_peered":"2024-02-10T05:55:03.129054+0000","last_unstale":"2024-02-10T05:55:36.014511+0000","last_undegraded":"2024-02-10T05:55:36.014511+0000","last_fullsized":"2024-02-10T05:55:36.014511+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:26:48.363844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.031585+0000","last_change":"2024-02-10T05:54:56.540743+0000","last_active":"2024-02-10T05:55:36.031585+0000","last_peered":"2024-02-10T05:55:36.031585+0000","last_clean":"2024-02-10T05:55:36.031585+0000","last_became_active":"2024-02-10T05:54:56.540470+0000","last_became_peered":"2024-02-10T05:54:56.540470+0000","last_unstale":"2024-02-10T05:55:36.031585+0000","last_undegraded":"2024-02-10T05:55:36.031585+0000","last_fullsized":"2024-02-10T05:55:36.031585+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:23:51.723375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"67'27","reported_seq":78,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014233+0000","last_change":"2024-02-10T05:54:58.557655+0000","last_active":"2024-02-10T05:55:36.014233+0000","last_peered":"2024-02-10T05:55:36.014233+0000","last_clean":"2024-02-10T05:55:36.014233+0000","last_became_active":"2024-02-10T05:54:58.557353+0000","last_became_peered":"2024-02-10T05:54:58.557353+0000","last_unstale":"2024-02-10T05:55:36.014233+0000","last_undegraded":"2024-02-10T05:55:36.014233+0000","last_fullsized":"2024-02-10T05:55:36.014233+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:53:54.536143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:56:06.597602+0000","last_change":"2024-02-10T05:55:00.577150+0000","last_active":"2024-02-10T05:56:06.597602+0000","last_peered":"2024-02-10T05:56:06.597602+0000","last_clean":"2024-02-10T05:56:06.597602+0000","last_became_active":"2024-02-10T05:55:00.576801+0000","last_became_peered":"2024-02-10T05:55:00.576801+0000","last_unstale":"2024-02-10T05:56:06.597602+0000","last_undegraded":"2024-02-10T05:56:06.597602+0000","last_fullsized":"2024-02-10T05:56:06.597602+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:56:06.372334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338372+0000","last_change":"2024-02-10T05:55:02.587625+0000","last_active":"2024-02-10T05:55:36.338372+0000","last_peered":"2024-02-10T05:55:36.338372+0000","last_clean":"2024-02-10T05:55:36.338372+0000","last_became_active":"2024-02-10T05:55:02.587320+0000","last_became_peered":"2024-02-10T05:55:02.587320+0000","last_unstale":"2024-02-10T05:55:36.338372+0000","last_undegraded":"2024-02-10T05:55:36.338372+0000","last_fullsized":"2024-02-10T05:55:36.338372+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:19:15.357080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.031567+0000","last_change":"2024-02-10T05:54:56.542102+0000","last_active":"2024-02-10T05:55:36.031567+0000","last_peered":"2024-02-10T05:55:36.031567+0000","last_clean":"2024-02-10T05:55:36.031567+0000","last_became_active":"2024-02-10T05:54:56.540632+0000","last_became_peered":"2024-02-10T05:54:56.540632+0000","last_unstale":"2024-02-10T05:55:36.031567+0000","last_undegraded":"2024-02-10T05:55:36.031567+0000","last_fullsized":"2024-02-10T05:55:36.031567+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:11:13.084896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"67'24","reported_seq":76,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.736682+0000","last_change":"2024-02-10T05:54:58.563244+0000","last_active":"2024-02-10T05:55:35.736682+0000","last_peered":"2024-02-10T05:55:35.736682+0000","last_clean":"2024-02-10T05:55:35.736682+0000","last_became_active":"2024-02-10T05:54:58.562959+0000","last_became_peered":"2024-02-10T05:54:58.562959+0000","last_unstale":"2024-02-10T05:55:35.736682+0000","last_undegraded":"2024-02-10T05:55:35.736682+0000","last_fullsized":"2024-02-10T05:55:35.736682+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:13:23.144091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.026823+0000","last_change":"2024-02-10T05:55:00.568890+0000","last_active":"2024-02-10T05:55:36.026823+0000","last_peered":"2024-02-10T05:55:36.026823+0000","last_clean":"2024-02-10T05:55:36.026823+0000","last_became_active":"2024-02-10T05:55:00.568490+0000","last_became_peered":"2024-02-10T05:55:00.568490+0000","last_unstale":"2024-02-10T05:55:36.026823+0000","last_undegraded":"2024-02-10T05:55:36.026823+0000","last_fullsized":"2024-02-10T05:55:36.026823+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:25:27.878666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.030788+0000","last_change":"2024-02-10T05:55:02.583935+0000","last_active":"2024-02-10T05:55:36.030788+0000","last_peered":"2024-02-10T05:55:36.030788+0000","last_clean":"2024-02-10T05:55:36.030788+0000","last_became_active":"2024-02-10T05:55:02.583552+0000","last_became_peered":"2024-02-10T05:55:02.583552+0000","last_unstale":"2024-02-10T05:55:36.030788+0000","last_undegraded":"2024-02-10T05:55:36.030788+0000","last_fullsized":"2024-02-10T05:55:36.030788+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:12:25.367129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.027312+0000","last_change":"2024-02-10T05:55:00.573921+0000","last_active":"2024-02-10T05:55:36.027312+0000","last_peered":"2024-02-10T05:55:36.027312+0000","last_clean":"2024-02-10T05:55:36.027312+0000","last_became_active":"2024-02-10T05:55:00.573600+0000","last_became_peered":"2024-02-10T05:55:00.573600+0000","last_unstale":"2024-02-10T05:55:36.027312+0000","last_undegraded":"2024-02-10T05:55:36.027312+0000","last_fullsized":"2024-02-10T05:55:36.027312+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:56:48.647371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.031561+0000","last_change":"2024-02-10T05:54:56.541434+0000","last_active":"2024-02-10T05:55:36.031561+0000","last_peered":"2024-02-10T05:55:36.031561+0000","last_clean":"2024-02-10T05:55:36.031561+0000","last_became_active":"2024-02-10T05:54:56.541103+0000","last_became_peered":"2024-02-10T05:54:56.541103+0000","last_unstale":"2024-02-10T05:55:36.031561+0000","last_undegraded":"2024-02-10T05:55:36.031561+0000","last_fullsized":"2024-02-10T05:55:36.031561+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:28:11.293509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"67'25","reported_seq":72,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338322+0000","last_change":"2024-02-10T05:54:58.560683+0000","last_active":"2024-02-10T05:55:36.338322+0000","last_peered":"2024-02-10T05:55:36.338322+0000","last_clean":"2024-02-10T05:55:36.338322+0000","last_became_active":"2024-02-10T05:54:58.560254+0000","last_became_peered":"2024-02-10T05:54:58.560254+0000","last_unstale":"2024-02-10T05:55:36.338322+0000","last_undegraded":"2024-02-10T05:55:36.338322+0000","last_fullsized":"2024-02-10T05:55:36.338322+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:44:10.134562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.738990+0000","last_change":"2024-02-10T05:55:02.582079+0000","last_active":"2024-02-10T05:55:35.738990+0000","last_peered":"2024-02-10T05:55:35.738990+0000","last_clean":"2024-02-10T05:55:35.738990+0000","last_became_active":"2024-02-10T05:55:02.581653+0000","last_became_peered":"2024-02-10T05:55:02.581653+0000","last_unstale":"2024-02-10T05:55:35.738990+0000","last_undegraded":"2024-02-10T05:55:35.738990+0000","last_fullsized":"2024-02-10T05:55:35.738990+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:13:58.090416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.027133+0000","last_change":"2024-02-10T05:55:00.576963+0000","last_active":"2024-02-10T05:55:36.027133+0000","last_peered":"2024-02-10T05:55:36.027133+0000","last_clean":"2024-02-10T05:55:36.027133+0000","last_became_active":"2024-02-10T05:55:00.576597+0000","last_became_peered":"2024-02-10T05:55:00.576597+0000","last_unstale":"2024-02-10T05:55:36.027133+0000","last_undegraded":"2024-02-10T05:55:36.027133+0000","last_fullsized":"2024-02-10T05:55:36.027133+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:41:11.237616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014603+0000","last_change":"2024-02-10T05:54:56.544756+0000","last_active":"2024-02-10T05:55:36.014603+0000","last_peered":"2024-02-10T05:55:36.014603+0000","last_clean":"2024-02-10T05:55:36.014603+0000","last_became_active":"2024-02-10T05:54:56.544460+0000","last_became_peered":"2024-02-10T05:54:56.544460+0000","last_unstale":"2024-02-10T05:55:36.014603+0000","last_undegraded":"2024-02-10T05:55:36.014603+0000","last_fullsized":"2024-02-10T05:55:36.014603+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:13:47.647366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"67'15","reported_seq":75,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.744406+0000","last_change":"2024-02-10T05:55:25.927808+0000","last_active":"2024-02-10T05:55:35.744406+0000","last_peered":"2024-02-10T05:55:35.744406+0000","last_clean":"2024-02-10T05:55:35.744406+0000","last_became_active":"2024-02-10T05:55:25.920369+0000","last_became_peered":"2024-02-10T05:55:25.920369+0000","last_unstale":"2024-02-10T05:55:35.744406+0000","last_undegraded":"2024-02-10T05:55:35.744406+0000","last_fullsized":"2024-02-10T05:55:35.744406+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:41:54.970006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338864+0000","last_change":"2024-02-10T05:55:02.581776+0000","last_active":"2024-02-10T05:55:36.338864+0000","last_peered":"2024-02-10T05:55:36.338864+0000","last_clean":"2024-02-10T05:55:36.338864+0000","last_became_active":"2024-02-10T05:55:02.581459+0000","last_became_peered":"2024-02-10T05:55:02.581459+0000","last_unstale":"2024-02-10T05:55:36.338864+0000","last_undegraded":"2024-02-10T05:55:36.338864+0000","last_fullsized":"2024-02-10T05:55:36.338864+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:34:16.027593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":42,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014361+0000","last_change":"2024-02-10T05:55:00.569960+0000","last_active":"2024-02-10T05:55:36.014361+0000","last_peered":"2024-02-10T05:55:36.014361+0000","last_clean":"2024-02-10T05:55:36.014361+0000","last_became_active":"2024-02-10T05:54:56.543145+0000","last_became_peered":"2024-02-10T05:54:56.543145+0000","last_unstale":"2024-02-10T05:55:36.014361+0000","last_undegraded":"2024-02-10T05:55:36.014361+0000","last_fullsized":"2024-02-10T05:55:36.014361+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:27:59.812610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047011520000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735203+0000","last_change":"2024-02-10T05:54:56.533400+0000","last_active":"2024-02-10T05:55:35.735203+0000","last_peered":"2024-02-10T05:55:35.735203+0000","last_clean":"2024-02-10T05:55:35.735203+0000","last_became_active":"2024-02-10T05:54:56.533110+0000","last_became_peered":"2024-02-10T05:54:56.533110+0000","last_unstale":"2024-02-10T05:55:35.735203+0000","last_undegraded":"2024-02-10T05:55:35.735203+0000","last_fullsized":"2024-02-10T05:55:35.735203+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:13:05.620672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"67'34","reported_seq":105,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:56:06.597383+0000","last_change":"2024-02-10T05:54:58.557933+0000","last_active":"2024-02-10T05:56:06.597383+0000","last_peered":"2024-02-10T05:56:06.597383+0000","last_clean":"2024-02-10T05:56:06.597383+0000","last_became_active":"2024-02-10T05:54:58.557563+0000","last_became_peered":"2024-02-10T05:54:58.557563+0000","last_unstale":"2024-02-10T05:56:06.597383+0000","last_undegraded":"2024-02-10T05:56:06.597383+0000","last_fullsized":"2024-02-10T05:56:06.597383+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:33:59.536452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.739962+0000","last_change":"2024-02-10T05:55:00.563519+0000","last_active":"2024-02-10T05:55:35.739962+0000","last_peered":"2024-02-10T05:55:35.739962+0000","last_clean":"2024-02-10T05:55:35.739962+0000","last_became_active":"2024-02-10T05:55:00.562557+0000","last_became_peered":"2024-02-10T05:55:00.562557+0000","last_unstale":"2024-02-10T05:55:35.739962+0000","last_undegraded":"2024-02-10T05:55:35.739962+0000","last_fullsized":"2024-02-10T05:55:35.739962+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:22:44.771980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.031638+0000","last_change":"2024-02-10T05:55:03.130189+0000","last_active":"2024-02-10T05:55:36.031638+0000","last_peered":"2024-02-10T05:55:36.031638+0000","last_clean":"2024-02-10T05:55:36.031638+0000","last_became_active":"2024-02-10T05:55:03.129911+0000","last_became_peered":"2024-02-10T05:55:03.129911+0000","last_unstale":"2024-02-10T05:55:36.031638+0000","last_undegraded":"2024-02-10T05:55:36.031638+0000","last_fullsized":"2024-02-10T05:55:36.031638+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:57:22.598979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.742524+0000","last_change":"2024-02-10T05:54:56.542933+0000","last_active":"2024-02-10T05:55:35.742524+0000","last_peered":"2024-02-10T05:55:35.742524+0000","last_clean":"2024-02-10T05:55:35.742524+0000","last_became_active":"2024-02-10T05:54:56.542659+0000","last_became_peered":"2024-02-10T05:54:56.542659+0000","last_unstale":"2024-02-10T05:55:35.742524+0000","last_undegraded":"2024-02-10T05:55:35.742524+0000","last_fullsized":"2024-02-10T05:55:35.742524+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:05:37.975285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.026992+0000","last_change":"2024-02-10T05:55:00.564438+0000","last_active":"2024-02-10T05:55:36.026992+0000","last_peered":"2024-02-10T05:55:36.026992+0000","last_clean":"2024-02-10T05:55:36.026992+0000","last_became_active":"2024-02-10T05:54:56.544669+0000","last_became_peered":"2024-02-10T05:54:56.544669+0000","last_unstale":"2024-02-10T05:55:36.026992+0000","last_undegraded":"2024-02-10T05:55:36.026992+0000","last_fullsized":"2024-02-10T05:55:36.026992+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:37:25.584726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012041790000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'19","reported_seq":27,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.027523+0000","last_change":"2024-02-10T05:55:25.931284+0000","last_active":"2024-02-10T05:55:36.027523+0000","last_peered":"2024-02-10T05:55:36.027523+0000","last_clean":"2024-02-10T05:55:36.027523+0000","last_became_active":"2024-02-10T05:55:25.921942+0000","last_became_peered":"2024-02-10T05:55:25.921942+0000","last_unstale":"2024-02-10T05:55:36.027523+0000","last_undegraded":"2024-02-10T05:55:36.027523+0000","last_fullsized":"2024-02-10T05:55:36.027523+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:12:33.977505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":26,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.027482+0000","last_change":"2024-02-10T05:55:00.572364+0000","last_active":"2024-02-10T05:55:36.027482+0000","last_peered":"2024-02-10T05:55:36.027482+0000","last_clean":"2024-02-10T05:55:36.027482+0000","last_became_active":"2024-02-10T05:55:00.571963+0000","last_became_peered":"2024-02-10T05:55:00.571963+0000","last_unstale":"2024-02-10T05:55:36.027482+0000","last_undegraded":"2024-02-10T05:55:36.027482+0000","last_fullsized":"2024-02-10T05:55:36.027482+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:13:12.833829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014859+0000","last_change":"2024-02-10T05:55:02.582765+0000","last_active":"2024-02-10T05:55:36.014859+0000","last_peered":"2024-02-10T05:55:36.014859+0000","last_clean":"2024-02-10T05:55:36.014859+0000","last_became_active":"2024-02-10T05:55:02.582256+0000","last_became_peered":"2024-02-10T05:55:02.582256+0000","last_unstale":"2024-02-10T05:55:36.014859+0000","last_undegraded":"2024-02-10T05:55:36.014859+0000","last_fullsized":"2024-02-10T05:55:36.014859+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:24:07.550336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"71'188","reported_seq":345,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:37.963163+0000","last_change":"2024-02-10T05:54:47.640507+0000","last_active":"2024-02-10T05:55:37.963163+0000","last_peered":"2024-02-10T05:55:37.963163+0000","last_clean":"2024-02-10T05:55:37.963163+0000","last_became_active":"2024-02-10T05:54:47.640071+0000","last_became_peered":"2024-02-10T05:54:47.640071+0000","last_unstale":"2024-02-10T05:55:37.963163+0000","last_undegraded":"2024-02-10T05:55:37.963163+0000","last_fullsized":"2024-02-10T05:55:37.963163+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:52:38.887992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:52:38.887992+0000","last_clean_scrub_stamp":"2024-02-10T05:52:38.887992+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:16:51.052806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338294+0000","last_change":"2024-02-10T05:54:56.543222+0000","last_active":"2024-02-10T05:55:36.338294+0000","last_peered":"2024-02-10T05:55:36.338294+0000","last_clean":"2024-02-10T05:55:36.338294+0000","last_became_active":"2024-02-10T05:54:56.542949+0000","last_became_peered":"2024-02-10T05:54:56.542949+0000","last_unstale":"2024-02-10T05:55:36.338294+0000","last_undegraded":"2024-02-10T05:55:36.338294+0000","last_fullsized":"2024-02-10T05:55:36.338294+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:18:35.117143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"67'16","reported_seq":67,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:56:06.597602+0000","last_change":"2024-02-10T05:54:58.562821+0000","last_active":"2024-02-10T05:56:06.597602+0000","last_peered":"2024-02-10T05:56:06.597602+0000","last_clean":"2024-02-10T05:56:06.597602+0000","last_became_active":"2024-02-10T05:54:58.562517+0000","last_became_peered":"2024-02-10T05:54:58.562517+0000","last_unstale":"2024-02-10T05:56:06.597602+0000","last_undegraded":"2024-02-10T05:56:06.597602+0000","last_fullsized":"2024-02-10T05:56:06.597602+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:38:12.263776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.031373+0000","last_change":"2024-02-10T05:55:00.574325+0000","last_active":"2024-02-10T05:55:36.031373+0000","last_peered":"2024-02-10T05:55:36.031373+0000","last_clean":"2024-02-10T05:55:36.031373+0000","last_became_active":"2024-02-10T05:55:00.574036+0000","last_became_peered":"2024-02-10T05:55:00.574036+0000","last_unstale":"2024-02-10T05:55:36.031373+0000","last_undegraded":"2024-02-10T05:55:36.031373+0000","last_fullsized":"2024-02-10T05:55:36.031373+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:28:14.174664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.743650+0000","last_change":"2024-02-10T05:55:02.586999+0000","last_active":"2024-02-10T05:55:35.743650+0000","last_peered":"2024-02-10T05:55:35.743650+0000","last_clean":"2024-02-10T05:55:35.743650+0000","last_became_active":"2024-02-10T05:55:02.586660+0000","last_became_peered":"2024-02-10T05:55:02.586660+0000","last_unstale":"2024-02-10T05:55:35.743650+0000","last_undegraded":"2024-02-10T05:55:35.743650+0000","last_fullsized":"2024-02-10T05:55:35.743650+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:17:05.193264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.739223+0000","last_change":"2024-02-10T05:54:56.544656+0000","last_active":"2024-02-10T05:55:35.739223+0000","last_peered":"2024-02-10T05:55:35.739223+0000","last_clean":"2024-02-10T05:55:35.739223+0000","last_became_active":"2024-02-10T05:54:56.544382+0000","last_became_peered":"2024-02-10T05:54:56.544382+0000","last_unstale":"2024-02-10T05:55:35.739223+0000","last_undegraded":"2024-02-10T05:55:35.739223+0000","last_fullsized":"2024-02-10T05:55:35.739223+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:16:42.905086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":112,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:56:10.551619+0000","last_change":"2024-02-10T05:55:00.566727+0000","last_active":"2024-02-10T05:56:10.551619+0000","last_peered":"2024-02-10T05:56:10.551619+0000","last_clean":"2024-02-10T05:56:10.551619+0000","last_became_active":"2024-02-10T05:54:56.539786+0000","last_became_peered":"2024-02-10T05:54:56.539786+0000","last_unstale":"2024-02-10T05:56:10.551619+0000","last_undegraded":"2024-02-10T05:56:10.551619+0000","last_fullsized":"2024-02-10T05:56:10.551619+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:17:41.995852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001442444,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":64,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":47,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.741150+0000","last_change":"2024-02-10T05:54:58.557165+0000","last_active":"2024-02-10T05:55:35.741150+0000","last_peered":"2024-02-10T05:55:35.741150+0000","last_clean":"2024-02-10T05:55:35.741150+0000","last_became_active":"2024-02-10T05:54:58.556344+0000","last_became_peered":"2024-02-10T05:54:58.556344+0000","last_unstale":"2024-02-10T05:55:35.741150+0000","last_undegraded":"2024-02-10T05:55:35.741150+0000","last_fullsized":"2024-02-10T05:55:35.741150+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:12:08.664273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.015042+0000","last_change":"2024-02-10T05:55:00.578052+0000","last_active":"2024-02-10T05:55:36.015042+0000","last_peered":"2024-02-10T05:55:36.015042+0000","last_clean":"2024-02-10T05:55:36.015042+0000","last_became_active":"2024-02-10T05:55:00.577752+0000","last_became_peered":"2024-02-10T05:55:00.577752+0000","last_unstale":"2024-02-10T05:55:36.015042+0000","last_undegraded":"2024-02-10T05:55:36.015042+0000","last_fullsized":"2024-02-10T05:55:36.015042+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:59:14.813996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.745494+0000","last_change":"2024-02-10T05:55:03.129468+0000","last_active":"2024-02-10T05:55:35.745494+0000","last_peered":"2024-02-10T05:55:35.745494+0000","last_clean":"2024-02-10T05:55:35.745494+0000","last_became_active":"2024-02-10T05:55:03.129074+0000","last_became_peered":"2024-02-10T05:55:03.129074+0000","last_unstale":"2024-02-10T05:55:35.745494+0000","last_undegraded":"2024-02-10T05:55:35.745494+0000","last_fullsized":"2024-02-10T05:55:35.745494+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:55:37.794146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.745373+0000","last_change":"2024-02-10T05:54:56.541048+0000","last_active":"2024-02-10T05:55:35.745373+0000","last_peered":"2024-02-10T05:55:35.745373+0000","last_clean":"2024-02-10T05:55:35.745373+0000","last_became_active":"2024-02-10T05:54:56.540749+0000","last_became_peered":"2024-02-10T05:54:56.540749+0000","last_unstale":"2024-02-10T05:55:35.745373+0000","last_undegraded":"2024-02-10T05:55:35.745373+0000","last_fullsized":"2024-02-10T05:55:35.745373+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:22:30.533260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"67'25","reported_seq":77,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.744112+0000","last_change":"2024-02-10T05:54:58.557311+0000","last_active":"2024-02-10T05:55:35.744112+0000","last_peered":"2024-02-10T05:55:35.744112+0000","last_clean":"2024-02-10T05:55:35.744112+0000","last_became_active":"2024-02-10T05:54:58.556986+0000","last_became_peered":"2024-02-10T05:54:58.556986+0000","last_unstale":"2024-02-10T05:55:35.744112+0000","last_undegraded":"2024-02-10T05:55:35.744112+0000","last_fullsized":"2024-02-10T05:55:35.744112+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:11:21.366052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338270+0000","last_change":"2024-02-10T05:55:00.576171+0000","last_active":"2024-02-10T05:55:36.338270+0000","last_peered":"2024-02-10T05:55:36.338270+0000","last_clean":"2024-02-10T05:55:36.338270+0000","last_became_active":"2024-02-10T05:55:00.575793+0000","last_became_peered":"2024-02-10T05:55:00.575793+0000","last_unstale":"2024-02-10T05:55:36.338270+0000","last_undegraded":"2024-02-10T05:55:36.338270+0000","last_fullsized":"2024-02-10T05:55:36.338270+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:33:05.098656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.738607+0000","last_change":"2024-02-10T05:55:02.584717+0000","last_active":"2024-02-10T05:55:35.738607+0000","last_peered":"2024-02-10T05:55:35.738607+0000","last_clean":"2024-02-10T05:55:35.738607+0000","last_became_active":"2024-02-10T05:55:02.584406+0000","last_became_peered":"2024-02-10T05:55:02.584406+0000","last_unstale":"2024-02-10T05:55:35.738607+0000","last_undegraded":"2024-02-10T05:55:35.738607+0000","last_fullsized":"2024-02-10T05:55:35.738607+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:05:36.016999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014722+0000","last_change":"2024-02-10T05:54:56.545279+0000","last_active":"2024-02-10T05:55:36.014722+0000","last_peered":"2024-02-10T05:55:36.014722+0000","last_clean":"2024-02-10T05:55:36.014722+0000","last_became_active":"2024-02-10T05:54:56.545023+0000","last_became_peered":"2024-02-10T05:54:56.545023+0000","last_unstale":"2024-02-10T05:55:36.014722+0000","last_undegraded":"2024-02-10T05:55:36.014722+0000","last_fullsized":"2024-02-10T05:55:36.014722+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:26:36.285352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"67'18","reported_seq":58,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.743139+0000","last_change":"2024-02-10T05:54:58.566635+0000","last_active":"2024-02-10T05:55:35.743139+0000","last_peered":"2024-02-10T05:55:35.743139+0000","last_clean":"2024-02-10T05:55:35.743139+0000","last_became_active":"2024-02-10T05:54:58.565164+0000","last_became_peered":"2024-02-10T05:54:58.565164+0000","last_unstale":"2024-02-10T05:55:35.743139+0000","last_undegraded":"2024-02-10T05:55:35.743139+0000","last_fullsized":"2024-02-10T05:55:35.743139+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:41:48.201138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:56:06.597371+0000","last_change":"2024-02-10T05:55:00.573505+0000","last_active":"2024-02-10T05:56:06.597371+0000","last_peered":"2024-02-10T05:56:06.597371+0000","last_clean":"2024-02-10T05:56:06.597371+0000","last_became_active":"2024-02-10T05:55:00.573222+0000","last_became_peered":"2024-02-10T05:55:00.573222+0000","last_unstale":"2024-02-10T05:56:06.597371+0000","last_undegraded":"2024-02-10T05:56:06.597371+0000","last_fullsized":"2024-02-10T05:56:06.597371+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:30:37.386873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.031370+0000","last_change":"2024-02-10T05:55:03.129638+0000","last_active":"2024-02-10T05:55:36.031370+0000","last_peered":"2024-02-10T05:55:36.031370+0000","last_clean":"2024-02-10T05:55:36.031370+0000","last_became_active":"2024-02-10T05:55:03.129290+0000","last_became_peered":"2024-02-10T05:55:03.129290+0000","last_unstale":"2024-02-10T05:55:36.031370+0000","last_undegraded":"2024-02-10T05:55:36.031370+0000","last_fullsized":"2024-02-10T05:55:36.031370+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:31:40.725143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.741295+0000","last_change":"2024-02-10T05:54:56.541481+0000","last_active":"2024-02-10T05:55:35.741295+0000","last_peered":"2024-02-10T05:55:35.741295+0000","last_clean":"2024-02-10T05:55:35.741295+0000","last_became_active":"2024-02-10T05:54:56.541170+0000","last_became_peered":"2024-02-10T05:54:56.541170+0000","last_unstale":"2024-02-10T05:55:35.741295+0000","last_undegraded":"2024-02-10T05:55:35.741295+0000","last_fullsized":"2024-02-10T05:55:35.741295+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:33:14.751742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"67'20","reported_seq":62,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.736547+0000","last_change":"2024-02-10T05:54:58.562058+0000","last_active":"2024-02-10T05:55:35.736547+0000","last_peered":"2024-02-10T05:55:35.736547+0000","last_clean":"2024-02-10T05:55:35.736547+0000","last_became_active":"2024-02-10T05:54:58.561057+0000","last_became_peered":"2024-02-10T05:54:58.561057+0000","last_unstale":"2024-02-10T05:55:35.736547+0000","last_undegraded":"2024-02-10T05:55:35.736547+0000","last_fullsized":"2024-02-10T05:55:35.736547+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:54:31.311039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:56:06.597417+0000","last_change":"2024-02-10T05:55:00.580085+0000","last_active":"2024-02-10T05:56:06.597417+0000","last_peered":"2024-02-10T05:56:06.597417+0000","last_clean":"2024-02-10T05:56:06.597417+0000","last_became_active":"2024-02-10T05:55:00.579395+0000","last_became_peered":"2024-02-10T05:55:00.579395+0000","last_unstale":"2024-02-10T05:56:06.597417+0000","last_undegraded":"2024-02-10T05:56:06.597417+0000","last_fullsized":"2024-02-10T05:56:06.597417+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:48:50.506985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735321+0000","last_change":"2024-02-10T05:55:02.584535+0000","last_active":"2024-02-10T05:55:35.735321+0000","last_peered":"2024-02-10T05:55:35.735321+0000","last_clean":"2024-02-10T05:55:35.735321+0000","last_became_active":"2024-02-10T05:55:02.583871+0000","last_became_peered":"2024-02-10T05:55:02.583871+0000","last_unstale":"2024-02-10T05:55:35.735321+0000","last_undegraded":"2024-02-10T05:55:35.735321+0000","last_fullsized":"2024-02-10T05:55:35.735321+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:54:30.587897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"60'1","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.339020+0000","last_change":"2024-02-10T05:54:56.543104+0000","last_active":"2024-02-10T05:55:36.339020+0000","last_peered":"2024-02-10T05:55:36.339020+0000","last_clean":"2024-02-10T05:55:36.339020+0000","last_became_active":"2024-02-10T05:54:56.542806+0000","last_became_peered":"2024-02-10T05:54:56.542806+0000","last_unstale":"2024-02-10T05:55:36.339020+0000","last_undegraded":"2024-02-10T05:55:36.339020+0000","last_fullsized":"2024-02-10T05:55:36.339020+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:12:06.292926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"67'30","reported_seq":85,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338143+0000","last_change":"2024-02-10T05:54:58.564802+0000","last_active":"2024-02-10T05:55:36.338143+0000","last_peered":"2024-02-10T05:55:36.338143+0000","last_clean":"2024-02-10T05:55:36.338143+0000","last_became_active":"2024-02-10T05:54:58.563860+0000","last_became_peered":"2024-02-10T05:54:58.563860+0000","last_unstale":"2024-02-10T05:55:36.338143+0000","last_undegraded":"2024-02-10T05:55:36.338143+0000","last_fullsized":"2024-02-10T05:55:36.338143+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:21:57.209980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735330+0000","last_change":"2024-02-10T05:55:00.579117+0000","last_active":"2024-02-10T05:55:35.735330+0000","last_peered":"2024-02-10T05:55:35.735330+0000","last_clean":"2024-02-10T05:55:35.735330+0000","last_became_active":"2024-02-10T05:55:00.578770+0000","last_became_peered":"2024-02-10T05:55:00.578770+0000","last_unstale":"2024-02-10T05:55:35.735330+0000","last_undegraded":"2024-02-10T05:55:35.735330+0000","last_fullsized":"2024-02-10T05:55:35.735330+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:35:37.992788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014575+0000","last_change":"2024-02-10T05:55:02.582138+0000","last_active":"2024-02-10T05:55:36.014575+0000","last_peered":"2024-02-10T05:55:36.014575+0000","last_clean":"2024-02-10T05:55:36.014575+0000","last_became_active":"2024-02-10T05:55:02.581817+0000","last_became_peered":"2024-02-10T05:55:02.581817+0000","last_unstale":"2024-02-10T05:55:36.014575+0000","last_undegraded":"2024-02-10T05:55:36.014575+0000","last_fullsized":"2024-02-10T05:55:36.014575+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:08:31.228173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:56:06.597628+0000","last_change":"2024-02-10T05:55:00.571661+0000","last_active":"2024-02-10T05:56:06.597628+0000","last_peered":"2024-02-10T05:56:06.597628+0000","last_clean":"2024-02-10T05:56:06.597628+0000","last_became_active":"2024-02-10T05:55:00.571303+0000","last_became_peered":"2024-02-10T05:55:00.571303+0000","last_unstale":"2024-02-10T05:56:06.597628+0000","last_undegraded":"2024-02-10T05:56:06.597628+0000","last_fullsized":"2024-02-10T05:56:06.597628+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:48:42.433150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338056+0000","last_change":"2024-02-10T05:54:56.542011+0000","last_active":"2024-02-10T05:55:36.338056+0000","last_peered":"2024-02-10T05:55:36.338056+0000","last_clean":"2024-02-10T05:55:36.338056+0000","last_became_active":"2024-02-10T05:54:56.541687+0000","last_became_peered":"2024-02-10T05:54:56.541687+0000","last_unstale":"2024-02-10T05:55:36.338056+0000","last_undegraded":"2024-02-10T05:55:36.338056+0000","last_fullsized":"2024-02-10T05:55:36.338056+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:55:43.359225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.745885+0000","last_change":"2024-02-10T05:54:58.558114+0000","last_active":"2024-02-10T05:55:35.745885+0000","last_peered":"2024-02-10T05:55:35.745885+0000","last_clean":"2024-02-10T05:55:35.745885+0000","last_became_active":"2024-02-10T05:54:58.557771+0000","last_became_peered":"2024-02-10T05:54:58.557771+0000","last_unstale":"2024-02-10T05:55:35.745885+0000","last_undegraded":"2024-02-10T05:55:35.745885+0000","last_fullsized":"2024-02-10T05:55:35.745885+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:03:19.565049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338454+0000","last_change":"2024-02-10T05:55:03.129563+0000","last_active":"2024-02-10T05:55:36.338454+0000","last_peered":"2024-02-10T05:55:36.338454+0000","last_clean":"2024-02-10T05:55:36.338454+0000","last_became_active":"2024-02-10T05:55:03.129023+0000","last_became_peered":"2024-02-10T05:55:03.129023+0000","last_unstale":"2024-02-10T05:55:36.338454+0000","last_undegraded":"2024-02-10T05:55:36.338454+0000","last_fullsized":"2024-02-10T05:55:36.338454+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:54:54.545671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.014988+0000","last_change":"2024-02-10T05:55:00.580520+0000","last_active":"2024-02-10T05:55:36.014988+0000","last_peered":"2024-02-10T05:55:36.014988+0000","last_clean":"2024-02-10T05:55:36.014988+0000","last_became_active":"2024-02-10T05:55:00.580196+0000","last_became_peered":"2024-02-10T05:55:00.580196+0000","last_unstale":"2024-02-10T05:55:36.014988+0000","last_undegraded":"2024-02-10T05:55:36.014988+0000","last_fullsized":"2024-02-10T05:55:36.014988+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:49:08.651272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735153+0000","last_change":"2024-02-10T05:54:56.534306+0000","last_active":"2024-02-10T05:55:35.735153+0000","last_peered":"2024-02-10T05:55:35.735153+0000","last_clean":"2024-02-10T05:55:35.735153+0000","last_became_active":"2024-02-10T05:54:56.533672+0000","last_became_peered":"2024-02-10T05:54:56.533672+0000","last_unstale":"2024-02-10T05:55:35.735153+0000","last_undegraded":"2024-02-10T05:55:35.735153+0000","last_fullsized":"2024-02-10T05:55:35.735153+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:28:23.020328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"67'17","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.735488+0000","last_change":"2024-02-10T05:54:58.563075+0000","last_active":"2024-02-10T05:55:35.735488+0000","last_peered":"2024-02-10T05:55:35.735488+0000","last_clean":"2024-02-10T05:55:35.735488+0000","last_became_active":"2024-02-10T05:54:58.562774+0000","last_became_peered":"2024-02-10T05:54:58.562774+0000","last_unstale":"2024-02-10T05:55:35.735488+0000","last_undegraded":"2024-02-10T05:55:35.735488+0000","last_fullsized":"2024-02-10T05:55:35.735488+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:00:33.210657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.027690+0000","last_change":"2024-02-10T05:55:02.583875+0000","last_active":"2024-02-10T05:55:36.027690+0000","last_peered":"2024-02-10T05:55:36.027690+0000","last_clean":"2024-02-10T05:55:36.027690+0000","last_became_active":"2024-02-10T05:55:02.583508+0000","last_became_peered":"2024-02-10T05:55:02.583508+0000","last_unstale":"2024-02-10T05:55:36.027690+0000","last_undegraded":"2024-02-10T05:55:36.027690+0000","last_fullsized":"2024-02-10T05:55:36.027690+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:23:35.028206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.745710+0000","last_change":"2024-02-10T05:55:00.570046+0000","last_active":"2024-02-10T05:55:35.745710+0000","last_peered":"2024-02-10T05:55:35.745710+0000","last_clean":"2024-02-10T05:55:35.745710+0000","last_became_active":"2024-02-10T05:55:00.569754+0000","last_became_peered":"2024-02-10T05:55:00.569754+0000","last_unstale":"2024-02-10T05:55:35.745710+0000","last_undegraded":"2024-02-10T05:55:35.745710+0000","last_fullsized":"2024-02-10T05:55:35.745710+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:09:55.813272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338108+0000","last_change":"2024-02-10T05:54:56.543143+0000","last_active":"2024-02-10T05:55:36.338108+0000","last_peered":"2024-02-10T05:55:36.338108+0000","last_clean":"2024-02-10T05:55:36.338108+0000","last_became_active":"2024-02-10T05:54:56.542841+0000","last_became_peered":"2024-02-10T05:54:56.542841+0000","last_unstale":"2024-02-10T05:55:36.338108+0000","last_undegraded":"2024-02-10T05:55:36.338108+0000","last_fullsized":"2024-02-10T05:55:36.338108+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:21:42.837802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":57,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.736194+0000","last_change":"2024-02-10T05:54:58.563683+0000","last_active":"2024-02-10T05:55:35.736194+0000","last_peered":"2024-02-10T05:55:35.736194+0000","last_clean":"2024-02-10T05:55:35.736194+0000","last_became_active":"2024-02-10T05:54:58.563288+0000","last_became_peered":"2024-02-10T05:54:58.563288+0000","last_unstale":"2024-02-10T05:55:35.736194+0000","last_undegraded":"2024-02-10T05:55:35.736194+0000","last_fullsized":"2024-02-10T05:55:35.736194+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:32:36.944544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.736016+0000","last_change":"2024-02-10T05:55:02.583004+0000","last_active":"2024-02-10T05:55:35.736016+0000","last_peered":"2024-02-10T05:55:35.736016+0000","last_clean":"2024-02-10T05:55:35.736016+0000","last_became_active":"2024-02-10T05:55:02.582694+0000","last_became_peered":"2024-02-10T05:55:02.582694+0000","last_unstale":"2024-02-10T05:55:35.736016+0000","last_undegraded":"2024-02-10T05:55:35.736016+0000","last_fullsized":"2024-02-10T05:55:35.736016+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:55:01.555329+0000","last_clean_scrub_stamp":"2024-02-10T05:55:01.555329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:44:46.960181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":52,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:56:06.597359+0000","last_change":"2024-02-10T05:55:00.574688+0000","last_active":"2024-02-10T05:56:06.597359+0000","last_peered":"2024-02-10T05:56:06.597359+0000","last_clean":"2024-02-10T05:56:06.597359+0000","last_became_active":"2024-02-10T05:55:00.574385+0000","last_became_peered":"2024-02-10T05:55:00.574385+0000","last_unstale":"2024-02-10T05:56:06.597359+0000","last_undegraded":"2024-02-10T05:56:06.597359+0000","last_fullsized":"2024-02-10T05:56:06.597359+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:59.544035+0000","last_clean_scrub_stamp":"2024-02-10T05:54:59.544035+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:56:06.166451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:36.338464+0000","last_change":"2024-02-10T05:54:56.549290+0000","last_active":"2024-02-10T05:55:36.338464+0000","last_peered":"2024-02-10T05:55:36.338464+0000","last_clean":"2024-02-10T05:55:36.338464+0000","last_became_active":"2024-02-10T05:54:56.548651+0000","last_became_peered":"2024-02-10T05:54:56.548651+0000","last_unstale":"2024-02-10T05:55:36.338464+0000","last_undegraded":"2024-02-10T05:55:36.338464+0000","last_fullsized":"2024-02-10T05:55:36.338464+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:55.517330+0000","last_clean_scrub_stamp":"2024-02-10T05:54:55.517330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:28:41.180202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-02-10T05:55:35.736041+0000","last_change":"2024-02-10T05:54:58.565766+0000","last_active":"2024-02-10T05:55:35.736041+0000","last_peered":"2024-02-10T05:55:35.736041+0000","last_clean":"2024-02-10T05:55:35.736041+0000","last_became_active":"2024-02-10T05:54:58.565487+0000","last_became_peered":"2024-02-10T05:54:58.565487+0000","last_unstale":"2024-02-10T05:55:35.736041+0000","last_undegraded":"2024-02-10T05:55:35.736041+0000","last_fullsized":"2024-02-10T05:55:35.736041+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:54:57.532742+0000","last_clean_scrub_stamp":"2024-02-10T05:54:57.532742+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:30:40.248722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":985,"num_read_kb":710,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":587,"ondisk_log_size":587,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":188,"ondisk_log_size":188,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":64,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234004,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32540,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708516,"statfs":{"total":95990841344,"available":95957520384,"internally_reserved":0,"allocated":1396736,"data_stored":923975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.59199999999999997}]},{"osd":1,"last update":"Sat Feb 10 05:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.61499999999999999}]},{"osd":2,"last update":"Sat Feb 10 05:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.49099999999999999}]},{"osd":3,"last update":"Sat Feb 10 05:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.53900000000000003}]},{"osd":4,"last update":"Sat Feb 10 05:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.371}]},{"osd":5,"last update":"Sat Feb 10 05:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.56100000000000005}]},{"osd":6,"last update":"Sat Feb 10 05:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.505},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.46400000000000002}]}]},{"osd":6,"up_from":46,"seq":197568495641,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":28356,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93712700,"statfs":{"total":95990841344,"available":95961804800,"internally_reserved":0,"allocated":1372160,"data_stored":922435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.65000000000000002}]},{"osd":1,"last update":"Sat Feb 10 05:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.629}]},{"osd":2,"last update":"Sat Feb 10 05:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.55900000000000005}]},{"osd":3,"last update":"Sat Feb 10 05:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.53600000000000003}]},{"osd":4,"last update":"Sat Feb 10 05:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.752}]},{"osd":5,"last update":"Sat Feb 10 05:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.67600000000000005}]},{"osd":7,"last update":"Sat Feb 10 05:55:47 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.71599999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31980,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709076,"statfs":{"total":95990841344,"available":95958093824,"internally_reserved":0,"allocated":819200,"data_stored":332904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.33800000000000002,"15min":0.308},"min":{"1min":0.248,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.33900000000000002,"15min":0.31},"min":{"1min":0.246,"5min":0.223,"15min":0.223},"max":{"1min":0.56000000000000005,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.32200000000000001}]},{"osd":2,"last update":"Sat Feb 10 05:55:42 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.37,"15min":0.35999999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.58599999999999997,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.40100000000000002,"15min":0.38700000000000001},"min":{"1min":0.27000000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.38700000000000001}]},{"osd":3,"last update":"Sat Feb 10 05:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.46300000000000002,"15min":0.46700000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.60199999999999998,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.44,"5min":0.45300000000000001,"15min":0.45500000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.35899999999999999}]},{"osd":4,"last update":"Sat Feb 10 05:55:34 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.621,"15min":0.628},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.81699999999999995,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.77400000000000002,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.56999999999999995}]},{"osd":5,"last update":"Sat Feb 10 05:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.61499999999999999,"15min":0.61699999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.751,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.57999999999999996,"15min":0.57699999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82899999999999996,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.60499999999999998}]},{"osd":6,"last update":"Sat Feb 10 05:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.623}]},{"osd":7,"last update":"Sat Feb 10 05:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":3.9510000000000001,"5min":3.9510000000000001,"15min":3.9510000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.66000000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32544,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708512,"statfs":{"total":95990841344,"available":95957516288,"internally_reserved":0,"allocated":1400832,"data_stored":924103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 05:55:14 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.34899999999999998,"15min":0.32500000000000001},"min":{"1min":0.25600000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.56200000000000006,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.36699999999999999,"15min":0.34599999999999997},"min":{"1min":0.24399999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.65700000000000003,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.375}]},{"osd":2,"last update":"Sat Feb 10 05:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.39000000000000001,"15min":0.36699999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.69899999999999995,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.38700000000000001,"15min":0.377},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.27100000000000002}]},{"osd":3,"last update":"Sat Feb 10 05:56:09 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.46500000000000002,"15min":0.46400000000000002},"min":{"1min":0.32600000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.45300000000000001,"15min":0.45600000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.59199999999999997,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.312}]},{"osd":4,"last update":"Sat Feb 10 05:55:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.60999999999999999,"15min":0.60599999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.83199999999999996,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.61799999999999999,"15min":0.61599999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.47199999999999998}]},{"osd":5,"last update":"Sat Feb 10 05:55:56 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.63700000000000001,"15min":0.63900000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.61599999999999999,"15min":0.61899999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.86199999999999999,"5min":0.879,"15min":0.879},"last":0.42199999999999999}]},{"osd":6,"last update":"Sat Feb 10 05:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.432},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.41199999999999998}]},{"osd":7,"last update":"Sat Feb 10 05:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.48699999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27792,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713264,"statfs":{"total":95990841344,"available":95962382336,"internally_reserved":0,"allocated":790528,"data_stored":332236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.40999999999999998,"15min":0.40200000000000002},"min":{"1min":0.26600000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.61799999999999999,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.57699999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.38600000000000001}]},{"osd":1,"last update":"Sat Feb 10 05:55:49 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.41999999999999998,"15min":0.42199999999999999},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.63700000000000001,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.41999999999999998,"15min":0.40999999999999998},"min":{"1min":0.26500000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.65400000000000003,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.34699999999999998}]},{"osd":3,"last update":"Sat Feb 10 05:55:10 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34999999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.501,"15min":0.499},"min":{"1min":0.35099999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.40699999999999997}]},{"osd":4,"last update":"Sat Feb 10 05:55:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58299999999999996,"15min":0.58199999999999996},"min":{"1min":0.41899999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.47799999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.82799999999999996}]},{"osd":5,"last update":"Sat Feb 10 05:55:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.64000000000000001,"15min":0.65000000000000002},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.78600000000000003,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.61099999999999999,"15min":0.622},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.73999999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.75900000000000001}]},{"osd":6,"last update":"Sat Feb 10 05:55:18 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.59999999999999998}]},{"osd":7,"last update":"Sat Feb 10 05:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.76900000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182440,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31976,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709080,"statfs":{"total":95990841344,"available":95958097920,"internally_reserved":0,"allocated":815104,"data_stored":333434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.42099999999999999,"15min":0.44},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.67600000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.42899999999999999,"15min":0.44500000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.54800000000000004}]},{"osd":1,"last update":"Sat Feb 10 05:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.45200000000000001,"15min":0.46999999999999997},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.58699999999999997,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.45600000000000002,"15min":0.47199999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.67000000000000004,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.501}]},{"osd":2,"last update":"Sat Feb 10 05:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.46200000000000002,"15min":0.47199999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.45200000000000001,"15min":0.46800000000000003},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.64500000000000002,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.41999999999999998}]},{"osd":4,"last update":"Sat Feb 10 05:55:36 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.58699999999999997,"15min":0.58399999999999996},"min":{"1min":0.44700000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.628,"5min":0.621,"15min":0.62},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.71499999999999997}]},{"osd":5,"last update":"Sat Feb 10 05:55:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.61899999999999999,"15min":0.625},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.623,"15min":0.621},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.78100000000000003}]},{"osd":6,"last update":"Sat Feb 10 05:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.72599999999999998}]},{"osd":7,"last update":"Sat Feb 10 05:55:45 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.88700000000000001}]}]},{"osd":4,"up_from":30,"seq":128849018915,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27808,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713248,"statfs":{"total":95990841344,"available":95962365952,"internally_reserved":0,"allocated":806912,"data_stored":332287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:55:35 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.54300000000000004,"15min":0.55100000000000005},"min":{"1min":0.38600000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.59399999999999997,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.437,"5min":0.377,"15min":0.377},"max":{"1min":0.68200000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.621}]},{"osd":1,"last update":"Sat Feb 10 05:55:35 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52500000000000002,"15min":0.53100000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.59699999999999998,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.53900000000000003,"15min":0.54400000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.64100000000000001,"5min":0.749,"15min":0.749},"last":0.80400000000000005}]},{"osd":2,"last update":"Sat Feb 10 05:55:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56399999999999995,"15min":0.56699999999999995},"min":{"1min":0.39600000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.67000000000000004,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.52600000000000002,"15min":0.53100000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.623,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.58699999999999997}]},{"osd":3,"last update":"Sat Feb 10 05:55:35 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.52600000000000002,"15min":0.53000000000000003},"min":{"1min":0.40799999999999997,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.68600000000000005,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.70799999999999996,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.55900000000000005}]},{"osd":5,"last update":"Sat Feb 10 05:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.432,"15min":0.438},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.56799999999999995,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.47699999999999998,"15min":0.47499999999999998},"min":{"1min":0.36399999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.71099999999999997}]},{"osd":6,"last update":"Sat Feb 10 05:55:19 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.77100000000000002}]},{"osd":7,"last update":"Sat Feb 10 05:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.69599999999999995}]}]},{"osd":5,"up_from":38,"seq":163208757278,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27788,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713268,"statfs":{"total":95990841344,"available":95962386432,"internally_reserved":0,"allocated":786432,"data_stored":332502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.55900000000000005,"15min":0.56699999999999995},"min":{"1min":0.35999999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.69999999999999996,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.73999999999999999,"5min":0.875,"15min":0.875},"last":0.54300000000000004}]},{"osd":1,"last update":"Sat Feb 10 05:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.747,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.55200000000000005,"15min":0.56200000000000006},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.63400000000000001,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.49399999999999999}]},{"osd":2,"last update":"Sat Feb 10 05:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.59799999999999998,"15min":0.61299999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.64600000000000002,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.64100000000000001,"15min":0.65600000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.82799999999999996,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":0.629}]},{"osd":3,"last update":"Sat Feb 10 05:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.44400000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.80900000000000005,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.58399999999999996,"15min":0.58999999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.67500000000000004,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.57099999999999995}]},{"osd":4,"last update":"Sat Feb 10 05:55:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39100000000000001,"15min":0.39000000000000001},"min":{"1min":0.30099999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.59799999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.38900000000000001,"15min":0.38500000000000001},"min":{"1min":0.28499999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.54800000000000004,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.47899999999999998}]},{"osd":6,"last update":"Sat Feb 10 05:55:20 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.42999999999999999}]},{"osd":7,"last update":"Sat Feb 10 05:55:44 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.374}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T05:56:18.658 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-10T05:56:18.658 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-10T05:56:18.658 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-10T05:56:18.659 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- ceph health --format=json 2024-02-10T05:56:18.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:18 smithi064 ceph-mon[25185]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:18.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:18 smithi064 ceph-mon[25185]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:56:18.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:18 smithi064 ceph-mon[38958]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:18.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:18 smithi064 ceph-mon[38958]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:56:18.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:18 smithi176 ceph-mon[30294]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:18.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:18 smithi176 ceph-mon[30294]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:56:18.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:56:18 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[66249]: debug there is no tcmu-runner data available 2024-02-10T05:56:21.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:20 smithi064 ceph-mon[25185]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:21.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:20 smithi064 ceph-mon[38958]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:21.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:20 smithi176 ceph-mon[30294]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:21.626 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:21 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: ::ffff:172.21.15.176 - - [10/Feb/2024:05:56:21] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T05:56:21.837 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:56:22.021 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:21 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:56:22.021 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:21 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:56:22.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:21 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:56:22.626 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:56:22 smithi064 systemd[1]: Starting Ceph alertmanager.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:56:23.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:22 smithi064 ceph-mon[38958]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:23.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:22 smithi064 ceph-mon[25185]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:23.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:22 smithi176 ceph-mon[30294]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:23.447 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:56:23 smithi064 podman[82833]: 2024-02-10 05:56:23.256551463 +0000 UTC m=+0.020430546 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-02-10T05:56:23.448 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:56:23 smithi064 podman[82833]: 2024-02-10 05:56:23.364499144 +0000 UTC m=+0.128378229 volume create 6c3e7da3029efa3cc91e36b12af7d16c0f61f96a31d201b0c6aba0107fca4f4d 2024-02-10T05:56:23.448 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:56:23 smithi064 podman[82833]: 2024-02-10 05:56:23.448057208 +0000 UTC m=+0.211936290 container create 0ddfabe698bcf5f4546ba362c4482b91cd36a9cb6c39d5e9c6650481b2bbf3c5 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T05:56:24.011 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:56:23 smithi064 podman[82833]: 2024-02-10 05:56:23.970051029 +0000 UTC m=+0.733930111 container init 0ddfabe698bcf5f4546ba362c4482b91cd36a9cb6c39d5e9c6650481b2bbf3c5 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T05:56:24.011 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:56:23 smithi064 podman[82833]: 2024-02-10 05:56:23.973276954 +0000 UTC m=+0.737156037 container start 0ddfabe698bcf5f4546ba362c4482b91cd36a9cb6c39d5e9c6650481b2bbf3c5 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T05:56:24.011 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:56:23 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-alertmanager-a[82864]: ts=2024-02-10T05:56:23.985Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-02-10T05:56:24.011 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:56:23 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-alertmanager-a[82864]: ts=2024-02-10T05:56:23.985Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-02-10T05:56:24.011 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:56:23 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-alertmanager-a[82864]: ts=2024-02-10T05:56:23.986Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.64 port=9094 2024-02-10T05:56:24.011 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:56:23 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-alertmanager-a[82864]: ts=2024-02-10T05:56:23.987Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-10T05:56:24.011 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:56:24 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-alertmanager-a[82864]: ts=2024-02-10T05:56:24.011Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T05:56:24.264 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:24 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: [10/Feb/2024:05:56:24] ENGINE Bus STOPPING 2024-02-10T05:56:24.264 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:56:24 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-alertmanager-a[82864]: ts=2024-02-10T05:56:24.011Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T05:56:24.264 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:56:24 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-alertmanager-a[82864]: ts=2024-02-10T05:56:24.014Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-02-10T05:56:24.265 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:56:24 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-alertmanager-a[82864]: ts=2024-02-10T05:56:24.014Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-02-10T05:56:24.265 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:56:24 smithi064 bash[82833]: 0ddfabe698bcf5f4546ba362c4482b91cd36a9cb6c39d5e9c6650481b2bbf3c5 2024-02-10T05:56:24.265 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:56:24 smithi064 systemd[1]: Started Ceph alertmanager.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:56:24.600 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-10T05:56:24.601 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-10T05:56:24.601 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:24 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: [10/Feb/2024:05:56:24] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T05:56:24.601 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:24 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: [10/Feb/2024:05:56:24] ENGINE Bus STOPPED 2024-02-10T05:56:24.601 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:24 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: [10/Feb/2024:05:56:24] ENGINE Bus STARTING 2024-02-10T05:56:24.601 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:24 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: [10/Feb/2024:05:56:24] ENGINE Serving on http://:::9283 2024-02-10T05:56:24.601 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:24 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: [10/Feb/2024:05:56:24] ENGINE Bus STARTED 2024-02-10T05:56:25.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[25185]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:25.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:25.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:25.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:25.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:25.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[25185]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-10T05:56:25.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:25.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:25.259 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:56:25.260 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[25185]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:56:25.260 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:25.260 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[25185]: Deploying daemon grafana.a on smithi176 2024-02-10T05:56:25.260 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[25185]: from='client.? 172.21.15.64:0/1761458119' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T05:56:25.260 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[38958]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:25.260 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:25.261 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:25.261 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:25.261 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:25.261 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[38958]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-10T05:56:25.261 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:25.261 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:25.261 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:56:25.261 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[38958]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:56:25.261 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:25.261 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[38958]: Deploying daemon grafana.a on smithi176 2024-02-10T05:56:25.261 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-mon[38958]: from='client.? 172.21.15.64:0/1761458119' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T05:56:25.286 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-10T05:56:25.287 INFO:tasks.cephadm:Setup complete, yielding 2024-02-10T05:56:25.287 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T05:56:25.297 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi064.front.sepia.ceph.com 2024-02-10T05:56:25.298 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-02-10T05:56:25.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:25 smithi176 ceph-mon[30294]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:25.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:25 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:25.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:25 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:25.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:25 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:25.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:25 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:25.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:25 smithi176 ceph-mon[30294]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-10T05:56:25.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:25 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:25.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:25 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:25.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:25 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:56:25.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:25 smithi176 ceph-mon[30294]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:56:25.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:25 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:25.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:25 smithi176 ceph-mon[30294]: Deploying daemon grafana.a on smithi176 2024-02-10T05:56:25.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:25 smithi176 ceph-mon[30294]: from='client.? 172.21.15.64:0/1761458119' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T05:56:25.543 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:56:26.376 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:56:25 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-alertmanager-a[82864]: ts=2024-02-10T05:56:25.988Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000788446s 2024-02-10T05:56:27.144 INFO:teuthology.orchestra.run.smithi064.stdout:167 167 2024-02-10T05:56:27.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:26 smithi176 ceph-mon[30294]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:27.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:26 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:27.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:26 smithi064 ceph-mon[25185]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:27.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:26 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:27.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:26 smithi064 ceph-mon[38958]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:27.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:26 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:27.899 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- bash -c 'ceph orch status' 2024-02-10T05:56:28.149 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:56:28.953 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:56:28 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[66249]: debug there is no tcmu-runner data available 2024-02-10T05:56:29.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:28 smithi176 ceph-mon[30294]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:29.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:28 smithi176 ceph-mon[30294]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:56:29.281 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:28 smithi064 ceph-mon[25185]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:29.281 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:29 smithi064 ceph-mon[25185]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:56:29.281 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:28 smithi064 ceph-mon[38958]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:29.282 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:29 smithi064 ceph-mon[38958]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:56:30.289 INFO:teuthology.orchestra.run.smithi064.stdout:Backend: cephadm 2024-02-10T05:56:30.289 INFO:teuthology.orchestra.run.smithi064.stdout:Available: Yes 2024-02-10T05:56:30.289 INFO:teuthology.orchestra.run.smithi064.stdout:Paused: No 2024-02-10T05:56:30.914 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- bash -c 'ceph orch ps' 2024-02-10T05:56:31.157 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:56:31.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:30 smithi176 ceph-mon[30294]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:31.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:30 smithi176 ceph-mon[30294]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:31.258 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:30 smithi064 ceph-mon[25185]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:31.258 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:30 smithi064 ceph-mon[25185]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:31.258 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:30 smithi064 ceph-mon[38958]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:31.259 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:30 smithi064 ceph-mon[38958]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:31.575 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:31 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: ::ffff:172.21.15.176 - - [10/Feb/2024:05:56:31] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T05:56:33.111 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:56:33.111 INFO:teuthology.orchestra.run.smithi064.stdout:alertmanager.a smithi064 *:9093,9094 starting - - - - 2024-02-10T05:56:33.111 INFO:teuthology.orchestra.run.smithi064.stdout:iscsi.iscsi.a smithi176 *:5000 running (85s) 53s ago 84s - - 3.6 c328ab9dcfc7 f7e7f53e9176 2024-02-10T05:56:33.111 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi176 *:8443,8765 running (5m) 53s ago 5m - - 19.0.0-1361-g290212a1 c328ab9dcfc7 0404dd01867c 2024-02-10T05:56:33.111 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.y smithi064 *:9283,8765 running (6m) 22s ago 6m 483M - 19.0.0-1361-g290212a1 c328ab9dcfc7 0209d0317adf 2024-02-10T05:56:33.111 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 running (7m) 22s ago 7m 46.4M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 22a0019eee34 2024-02-10T05:56:33.111 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi176 running (5m) 53s ago 5m - 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 87e8c2896dc7 2024-02-10T05:56:33.111 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi064 running (5m) 22s ago 5m 38.2M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 de9d44238760 2024-02-10T05:56:33.111 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (4m) 22s ago 4m 53.2M 3760M 19.0.0-1361-g290212a1 c328ab9dcfc7 c5e21e6393d6 2024-02-10T05:56:33.112 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi064 running (4m) 22s ago 4m 48.0M 3760M 19.0.0-1361-g290212a1 c328ab9dcfc7 8b4cf1877c69 2024-02-10T05:56:33.112 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi064 running (4m) 22s ago 4m 42.6M 3760M 19.0.0-1361-g290212a1 c328ab9dcfc7 3652b514b479 2024-02-10T05:56:33.112 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi064 running (3m) 22s ago 3m 54.1M 3760M 19.0.0-1361-g290212a1 c328ab9dcfc7 0b10f0df1984 2024-02-10T05:56:33.112 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi176 running (3m) 53s ago 3m - 3760M 19.0.0-1361-g290212a1 c328ab9dcfc7 9b73b664d9b5 2024-02-10T05:56:33.112 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi176 running (2m) 53s ago 2m - 3760M 19.0.0-1361-g290212a1 c328ab9dcfc7 b5886e6e8aec 2024-02-10T05:56:33.115 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi176 running (2m) 53s ago 2m - 3760M 19.0.0-1361-g290212a1 c328ab9dcfc7 b5681b7c4bee 2024-02-10T05:56:33.115 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi176 running (115s) 53s ago 114s - 3760M 19.0.0-1361-g290212a1 c328ab9dcfc7 9a0aa9ba5598 2024-02-10T05:56:33.115 INFO:teuthology.orchestra.run.smithi064.stdout:prometheus.a smithi176 *:9095 running (66s) 53s ago 65s - - 2.43.0 a07b618ecd1d e2ae1af39996 2024-02-10T05:56:33.115 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.a smithi064 *:80 running (98s) 22s ago 97s 76.3M - 19.0.0-1361-g290212a1 c328ab9dcfc7 760980f4d005 2024-02-10T05:56:33.125 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:32 smithi064 ceph-mon[25185]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:33.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:32 smithi064 ceph-mon[38958]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:33.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:32 smithi176 ceph-mon[30294]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:33.873 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- bash -c 'ceph orch ls' 2024-02-10T05:56:34.122 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:56:34.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:33 smithi064 ceph-mon[25185]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:34.126 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:56:33 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-alertmanager-a[82864]: ts=2024-02-10T05:56:33.990Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002626962s 2024-02-10T05:56:34.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:33 smithi064 ceph-mon[38958]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:34.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:33 smithi176 ceph-mon[30294]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:35.238 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:34 smithi176 ceph-mon[30294]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:35.277 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:34 smithi064 ceph-mon[25185]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:35.277 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:34 smithi064 ceph-mon[38958]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:36.049 INFO:teuthology.orchestra.run.smithi064.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-02-10T05:56:36.049 INFO:teuthology.orchestra.run.smithi064.stdout:alertmanager ?:9093,9094 0/1 - 81s smithi064=a;count:1 2024-02-10T05:56:36.049 INFO:teuthology.orchestra.run.smithi064.stdout:grafana ?:3000 0/1 - 78s smithi176=a;count:1 2024-02-10T05:56:36.049 INFO:teuthology.orchestra.run.smithi064.stdout:iscsi.datapool ?:5000 1/1 56s ago 92s smithi176=iscsi.a;count:1 2024-02-10T05:56:36.049 INFO:teuthology.orchestra.run.smithi064.stdout:mgr 2/2 56s ago 5m smithi064=y;smithi176=x;count:2 2024-02-10T05:56:36.049 INFO:teuthology.orchestra.run.smithi064.stdout:mon 3/3 56s ago 5m smithi064:172.21.15.64=a;smithi064:[v2:172.21.15.64:3301,v1:172.21.15.64:6790]=c;smithi176:172.21.15.176=b;count:3 2024-02-10T05:56:36.049 INFO:teuthology.orchestra.run.smithi064.stdout:node-exporter ?:9100 0/2 - 85s smithi064=a;smithi176=b;count:2 2024-02-10T05:56:36.049 INFO:teuthology.orchestra.run.smithi064.stdout:osd 8 56s ago - 2024-02-10T05:56:36.050 INFO:teuthology.orchestra.run.smithi064.stdout:prometheus ?:9095 1/1 56s ago 88s smithi176=a;count:1 2024-02-10T05:56:36.050 INFO:teuthology.orchestra.run.smithi064.stdout:rgw.foo.a ?:80 1/1 25s ago 100s smithi064=foo.a;count:1 2024-02-10T05:56:36.746 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-02-10T05:56:36.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:36 smithi064 ceph-mon[25185]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:36.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:36 smithi064 ceph-mon[25185]: from='client.24623 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:36.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:36 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:56:36.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:36 smithi064 ceph-mon[38958]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:36.876 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:36 smithi064 ceph-mon[38958]: from='client.24623 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:36.877 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:36 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:56:36.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:36 smithi176 ceph-mon[30294]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:36.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:36 smithi176 ceph-mon[30294]: from='client.24623 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:36.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:36 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:56:36.989 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:56:38.739 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:56:38 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[66249]: debug there is no tcmu-runner data available 2024-02-10T05:56:38.740 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:38 smithi176 systemd[1]: Starting Ceph grafana.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:56:38.793 INFO:teuthology.orchestra.run.smithi064.stdout:HOST ADDR LABELS STATUS 2024-02-10T05:56:38.793 INFO:teuthology.orchestra.run.smithi064.stdout:smithi064 172.21.15.64 2024-02-10T05:56:38.793 INFO:teuthology.orchestra.run.smithi064.stdout:smithi176 172.21.15.176 2024-02-10T05:56:38.793 INFO:teuthology.orchestra.run.smithi064.stdout:2 hosts in cluster 2024-02-10T05:56:39.065 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:38 smithi176 ceph-mon[30294]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:39.066 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:38 smithi176 podman[69714]: 2024-02-10 05:56:38.69081198 +0000 UTC m=+0.017727081 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-02-10T05:56:39.066 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:38 smithi176 podman[69714]: 2024-02-10 05:56:38.813090588 +0000 UTC m=+0.140005745 container create 1160ad9b2c7638c38d5c09ffd4388bd57c983cb67bcdb40d9a39f7443734b6a8 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a, maintainer=Paul Cuzner , architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, name=ubi8, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, io.buildah.version=1.32.0) 2024-02-10T05:56:39.066 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:38 smithi176 podman[69714]: 2024-02-10 05:56:38.981956516 +0000 UTC m=+0.308871625 container init 1160ad9b2c7638c38d5c09ffd4388bd57c983cb67bcdb40d9a39f7443734b6a8 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , io.buildah.version=1.32.0, description=Ceph Grafana Container, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, name=ubi8, vcs-type=git, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, io.openshift.expose-services=, build-date=2022-04-29T11:48:11.055897, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-02-10T05:56:39.066 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:38 smithi176 podman[69714]: 2024-02-10 05:56:38.984385979 +0000 UTC m=+0.311301080 container start 1160ad9b2c7638c38d5c09ffd4388bd57c983cb67bcdb40d9a39f7443734b6a8 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, version=8.5, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, release=239.1651231664, description=Ceph Grafana Container, vendor=Red Hat, Inc., io.buildah.version=1.32.0, io.openshift.tags=base rhel8, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-04-29T11:48:11.055897, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=) 2024-02-10T05:56:39.066 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 bash[69714]: 1160ad9b2c7638c38d5c09ffd4388bd57c983cb67bcdb40d9a39f7443734b6a8 2024-02-10T05:56:39.126 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:38 smithi064 ceph-mon[25185]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:39.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:38 smithi064 ceph-mon[38958]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:39.318 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 systemd[1]: Started Ceph grafana.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:56:39.318 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=settings t=2024-02-10T05:56:39.137195066Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-02-10T05:56:39.318 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=settings t=2024-02-10T05:56:39.137408536Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-02-10T05:56:39.318 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=settings t=2024-02-10T05:56:39.137431629Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-02-10T05:56:39.319 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=settings t=2024-02-10T05:56:39.137444458Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-02-10T05:56:39.319 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=settings t=2024-02-10T05:56:39.137460807Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-10T05:56:39.319 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=settings t=2024-02-10T05:56:39.137473876Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-10T05:56:39.319 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=settings t=2024-02-10T05:56:39.137484297Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-10T05:56:39.319 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=settings t=2024-02-10T05:56:39.137494914Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-10T05:56:39.319 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=settings t=2024-02-10T05:56:39.137505162Z level=info msg="Path Home" path=/usr/share/grafana 2024-02-10T05:56:39.320 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=settings t=2024-02-10T05:56:39.137515108Z level=info msg="Path Data" path=/var/lib/grafana 2024-02-10T05:56:39.320 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=settings t=2024-02-10T05:56:39.137524768Z level=info msg="Path Logs" path=/var/log/grafana 2024-02-10T05:56:39.320 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=settings t=2024-02-10T05:56:39.137534472Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-02-10T05:56:39.320 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=settings t=2024-02-10T05:56:39.137563345Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-02-10T05:56:39.320 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=settings t=2024-02-10T05:56:39.137574731Z level=info msg="App mode production" 2024-02-10T05:56:39.320 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=sqlstore t=2024-02-10T05:56:39.13764376Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-02-10T05:56:39.321 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=sqlstore t=2024-02-10T05:56:39.137666492Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-02-10T05:56:39.321 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.138647854Z level=info msg="Starting DB migrations" 2024-02-10T05:56:39.321 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.138734788Z level=info msg="Executing migration" id="create migration_log table" 2024-02-10T05:56:39.321 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.139273154Z level=info msg="Executing migration" id="create user table" 2024-02-10T05:56:39.321 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.139745549Z level=info msg="Executing migration" id="add unique index user.login" 2024-02-10T05:56:39.321 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.140235431Z level=info msg="Executing migration" id="add unique index user.email" 2024-02-10T05:56:39.322 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.140702197Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-02-10T05:56:39.322 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.141324932Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-02-10T05:56:39.322 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.141732381Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-02-10T05:56:39.322 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.142482522Z level=info msg="Executing migration" id="create user table v2" 2024-02-10T05:56:39.322 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.142887511Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-02-10T05:56:39.322 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.14331779Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-02-10T05:56:39.323 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.143798931Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-02-10T05:56:39.323 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.144186698Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-02-10T05:56:39.323 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.14456154Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-02-10T05:56:39.323 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.144959638Z level=info msg="Executing migration" id="Update user table charset" 2024-02-10T05:56:39.323 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.145274844Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-02-10T05:56:39.323 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.145663611Z level=info msg="Executing migration" id="Add missing user data" 2024-02-10T05:56:39.324 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.146053114Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-02-10T05:56:39.324 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.146477381Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-02-10T05:56:39.324 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.146861696Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-02-10T05:56:39.324 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.147296004Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-02-10T05:56:39.324 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.148407477Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-02-10T05:56:39.325 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.148863367Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-02-10T05:56:39.325 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.149314513Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-02-10T05:56:39.325 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.149711084Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-02-10T05:56:39.325 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.150125165Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-02-10T05:56:39.325 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.150553667Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-02-10T05:56:39.325 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.150822816Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-02-10T05:56:39.326 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.151200573Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-02-10T05:56:39.326 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.15156979Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-02-10T05:56:39.326 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.151919006Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-02-10T05:56:39.326 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.152292569Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-02-10T05:56:39.326 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.15290705Z level=info msg="Executing migration" id="create temp_user v2" 2024-02-10T05:56:39.326 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.15332645Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-02-10T05:56:39.326 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.153708772Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-02-10T05:56:39.326 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.154125308Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-02-10T05:56:39.327 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.154522289Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-02-10T05:56:39.327 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.154944236Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-02-10T05:56:39.327 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.155352519Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-02-10T05:56:39.327 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.156176749Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-02-10T05:56:39.327 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.15652531Z level=info msg="Executing migration" id="create star table" 2024-02-10T05:56:39.329 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.156962018Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-02-10T05:56:39.329 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.157394533Z level=info msg="Executing migration" id="create org table v1" 2024-02-10T05:56:39.329 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.157808083Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-02-10T05:56:39.329 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.158646474Z level=info msg="Executing migration" id="create org_user table v1" 2024-02-10T05:56:39.329 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.159626501Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-02-10T05:56:39.330 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.16015085Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-02-10T05:56:39.330 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.161179879Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-02-10T05:56:39.330 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.161783452Z level=info msg="Executing migration" id="Update org table charset" 2024-02-10T05:56:39.330 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.162647654Z level=info msg="Executing migration" id="Update org_user table charset" 2024-02-10T05:56:39.330 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.162955291Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-02-10T05:56:39.330 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.16397145Z level=info msg="Executing migration" id="create dashboard table" 2024-02-10T05:56:39.330 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.164682355Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-02-10T05:56:39.331 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.165511769Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-02-10T05:56:39.331 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.165974015Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-02-10T05:56:39.331 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.16697369Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-02-10T05:56:39.331 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.16767509Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-02-10T05:56:39.331 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.16851835Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-02-10T05:56:39.331 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.169435898Z level=info msg="Executing migration" id="create dashboard v2" 2024-02-10T05:56:39.332 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.170269087Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-02-10T05:56:39.332 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.171215643Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-02-10T05:56:39.332 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.17192552Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-02-10T05:56:39.332 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.172299199Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-02-10T05:56:39.332 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.175092415Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-02-10T05:56:39.332 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.175411354Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-02-10T05:56:39.333 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.177884644Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-02-10T05:56:39.333 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.178421285Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-02-10T05:56:39.333 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.180549404Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-02-10T05:56:39.333 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.18109278Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-02-10T05:56:39.333 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.181627875Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-02-10T05:56:39.333 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.184324688Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-02-10T05:56:39.334 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.187242311Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-02-10T05:56:39.334 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.190100963Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-02-10T05:56:39.334 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.19041266Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-02-10T05:56:39.334 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.193218511Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-02-10T05:56:39.334 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.196095447Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-02-10T05:56:39.334 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.198976821Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-02-10T05:56:39.334 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.199502526Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-02-10T05:56:39.335 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.202235799Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-02-10T05:56:39.335 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.202694017Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-02-10T05:56:39.335 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.204940617Z level=info msg="Executing migration" id="Update dashboard title length" 2024-02-10T05:56:39.335 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.207513625Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-02-10T05:56:39.335 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.208000439Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-02-10T05:56:39.335 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.210849496Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-02-10T05:56:39.336 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.213553998Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-02-10T05:56:39.336 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.21619137Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-02-10T05:56:39.336 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.216711023Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-02-10T05:56:39.336 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.219430337Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-02-10T05:56:39.336 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.22205137Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-02-10T05:56:39.337 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.224612231Z level=info msg="Executing migration" id="Add check_sum column" 2024-02-10T05:56:39.337 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.225170917Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-02-10T05:56:39.337 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.227707945Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-02-10T05:56:39.337 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.230341174Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-02-10T05:56:39.337 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.233047549Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-02-10T05:56:39.337 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.233516428Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-02-10T05:56:39.337 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.236229027Z level=info msg="Executing migration" id="create data_source table" 2024-02-10T05:56:39.337 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.238703534Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-02-10T05:56:39.337 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.239407583Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-02-10T05:56:39.337 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.240026886Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-02-10T05:56:39.337 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.240891839Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-02-10T05:56:39.337 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.241568811Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-02-10T05:56:39.337 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.242598328Z level=info msg="Executing migration" id="create data_source table v2" 2024-02-10T05:56:39.338 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.24301591Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-02-10T05:56:39.338 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.2438203Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-02-10T05:56:39.338 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.244623031Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-02-10T05:56:39.338 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.245511159Z level=info msg="Executing migration" id="Add column with_credentials" 2024-02-10T05:56:39.338 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.246170218Z level=info msg="Executing migration" id="Add secure json data column" 2024-02-10T05:56:39.338 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.246725011Z level=info msg="Executing migration" id="Update data_source table charset" 2024-02-10T05:56:39.338 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.247485896Z level=info msg="Executing migration" id="Update initial version to 1" 2024-02-10T05:56:39.338 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.247831712Z level=info msg="Executing migration" id="Add read_only data column" 2024-02-10T05:56:39.338 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.24837362Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-02-10T05:56:39.338 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.249261163Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-02-10T05:56:39.338 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.249743398Z level=info msg="Executing migration" id="Add uid column" 2024-02-10T05:56:39.338 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.250281016Z level=info msg="Executing migration" id="Update uid value" 2024-02-10T05:56:39.338 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.251168266Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-02-10T05:56:39.338 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.252067123Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-02-10T05:56:39.339 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.252795051Z level=info msg="Executing migration" id="create api_key table" 2024-02-10T05:56:39.339 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.25321062Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-02-10T05:56:39.339 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.25417131Z level=info msg="Executing migration" id="add index api_key.key" 2024-02-10T05:56:39.339 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.25493186Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-02-10T05:56:39.339 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.255809283Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-02-10T05:56:39.339 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.256234907Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-02-10T05:56:39.339 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.257058867Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-02-10T05:56:39.339 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.257969156Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-02-10T05:56:39.339 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.259113217Z level=info msg="Executing migration" id="create api_key table v2" 2024-02-10T05:56:39.339 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.260103976Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-02-10T05:56:39.339 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.260741892Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-02-10T05:56:39.339 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.261708036Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-02-10T05:56:39.339 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.262525377Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-02-10T05:56:39.339 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.263523976Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-02-10T05:56:39.340 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.264369852Z level=info msg="Executing migration" id="Update api_key table charset" 2024-02-10T05:56:39.340 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.265024176Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-02-10T05:56:39.340 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.267844774Z level=info msg="Executing migration" id="Add service account foreign key" 2024-02-10T05:56:39.340 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.270606184Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-02-10T05:56:39.340 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.273352908Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-02-10T05:56:39.340 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.276115571Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-02-10T05:56:39.340 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.278820758Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-02-10T05:56:39.340 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.281675211Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-02-10T05:56:39.340 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.284341498Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-02-10T05:56:39.340 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.2870069Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-02-10T05:56:39.340 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.289746677Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-02-10T05:56:39.341 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.292623626Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-02-10T05:56:39.341 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.295587862Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-02-10T05:56:39.341 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.298090913Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-02-10T05:56:39.341 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.300800173Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-02-10T05:56:39.341 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.303724682Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-02-10T05:56:39.341 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.306596823Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-02-10T05:56:39.341 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.309406722Z level=info msg="Executing migration" id="create quota table v1" 2024-02-10T05:56:39.341 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.311818799Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-02-10T05:56:39.341 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.314557813Z level=info msg="Executing migration" id="Update quota table charset" 2024-02-10T05:56:39.341 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.317327655Z level=info msg="Executing migration" id="create plugin_setting table" 2024-02-10T05:56:39.341 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.31996098Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-02-10T05:56:39.535 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-02-10T05:56:39.567 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.322587286Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-02-10T05:56:39.568 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.32513201Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-02-10T05:56:39.568 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.327955546Z level=info msg="Executing migration" id="create session table" 2024-02-10T05:56:39.568 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.330621783Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-02-10T05:56:39.568 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.333333688Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-02-10T05:56:39.568 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.336017982Z level=info msg="Executing migration" id="create playlist table v2" 2024-02-10T05:56:39.568 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.33886494Z level=info msg="Executing migration" id="create playlist item table v2" 2024-02-10T05:56:39.568 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.341610924Z level=info msg="Executing migration" id="Update playlist table charset" 2024-02-10T05:56:39.568 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.344254764Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-02-10T05:56:39.568 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.346789416Z level=info msg="Executing migration" id="drop preferences table v2" 2024-02-10T05:56:39.568 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.349559965Z level=info msg="Executing migration" id="drop preferences table v3" 2024-02-10T05:56:39.569 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.352315977Z level=info msg="Executing migration" id="create preferences table v3" 2024-02-10T05:56:39.569 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.355033709Z level=info msg="Executing migration" id="Update preferences table charset" 2024-02-10T05:56:39.569 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.357623256Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-02-10T05:56:39.569 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.360165808Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-02-10T05:56:39.569 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.361125329Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-02-10T05:56:39.569 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.361925823Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-02-10T05:56:39.569 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.362694731Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-02-10T05:56:39.569 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.363536445Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-02-10T05:56:39.569 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.364013888Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-02-10T05:56:39.569 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.364509596Z level=info msg="Executing migration" id="create alert table v1" 2024-02-10T05:56:39.569 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.36527689Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-02-10T05:56:39.569 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.367776567Z level=info msg="Executing migration" id="add index alert state" 2024-02-10T05:56:39.569 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.370465628Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-02-10T05:56:39.570 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.371022669Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-02-10T05:56:39.570 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.37142429Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-02-10T05:56:39.570 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.371859106Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-02-10T05:56:39.570 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.372272163Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-02-10T05:56:39.570 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.373702039Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-02-10T05:56:39.570 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.374104447Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-02-10T05:56:39.570 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.374525246Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-02-10T05:56:39.570 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.374820203Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-02-10T05:56:39.570 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.375195445Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-02-10T05:56:39.570 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.375579955Z level=info msg="Executing migration" id="Add column is_default" 2024-02-10T05:56:39.570 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.376203015Z level=info msg="Executing migration" id="Add column frequency" 2024-02-10T05:56:39.570 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.376826417Z level=info msg="Executing migration" id="Add column send_reminder" 2024-02-10T05:56:39.570 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.377460149Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-02-10T05:56:39.570 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.37811764Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-02-10T05:56:39.572 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.378543594Z level=info msg="Executing migration" id="Update alert table charset" 2024-02-10T05:56:39.572 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.378806219Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-02-10T05:56:39.572 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.379109017Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-02-10T05:56:39.572 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.379500104Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-02-10T05:56:39.572 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.379939162Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-02-10T05:56:39.572 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.380321634Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-02-10T05:56:39.572 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.380683551Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-02-10T05:56:39.572 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.381103049Z level=info msg="Executing migration" id="Add for to alert table" 2024-02-10T05:56:39.572 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.381727101Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-02-10T05:56:39.572 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.3823817Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-02-10T05:56:39.573 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.38269658Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-02-10T05:56:39.573 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.383138807Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-02-10T05:56:39.573 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.383536579Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-02-10T05:56:39.573 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.384199852Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-02-10T05:56:39.573 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.384503013Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-02-10T05:56:39.573 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.384888613Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-02-10T05:56:39.573 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.385306953Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-02-10T05:56:39.573 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.3855975Z level=info msg="Executing migration" id="create annotation table v5" 2024-02-10T05:56:39.573 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.38604692Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-02-10T05:56:39.573 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.388839121Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-02-10T05:56:39.573 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.389460082Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-02-10T05:56:39.573 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.38987181Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-02-10T05:56:39.573 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.390346813Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-02-10T05:56:39.574 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.392975304Z level=info msg="Executing migration" id="Update annotation table charset" 2024-02-10T05:56:39.574 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.393276869Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-02-10T05:56:39.574 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.395704913Z level=info msg="Executing migration" id="Drop category_id index" 2024-02-10T05:56:39.574 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.396227458Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-02-10T05:56:39.574 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.398453335Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-02-10T05:56:39.574 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.40114146Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-02-10T05:56:39.574 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.403906866Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-02-10T05:56:39.574 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.406651595Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-02-10T05:56:39.574 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.409512473Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-02-10T05:56:39.574 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.412077032Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-02-10T05:56:39.574 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.415031518Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-02-10T05:56:39.574 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.417527007Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-02-10T05:56:39.574 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.420307037Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-02-10T05:56:39.574 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.423016602Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-02-10T05:56:39.575 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.425767686Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-02-10T05:56:39.575 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.428326287Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-02-10T05:56:39.575 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.431078312Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-02-10T05:56:39.575 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.433827222Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-02-10T05:56:39.575 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.436516608Z level=info msg="Executing migration" id="Add epoch_end column" 2024-02-10T05:56:39.575 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.439216577Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-02-10T05:56:39.575 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.441918632Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-02-10T05:56:39.575 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.444332135Z level=info msg="Executing migration" id="Move region to single row" 2024-02-10T05:56:39.575 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.447123534Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-02-10T05:56:39.575 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.44974081Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-02-10T05:56:39.575 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.452539199Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-02-10T05:56:39.575 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.453282564Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-02-10T05:56:39.575 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.454118283Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-02-10T05:56:39.576 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.454922699Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-02-10T05:56:39.576 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.455789325Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-02-10T05:56:39.576 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.456564036Z level=info msg="Executing migration" id="create test_data table" 2024-02-10T05:56:39.576 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.456976833Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-02-10T05:56:39.576 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.457396964Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-02-10T05:56:39.576 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.458295276Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-02-10T05:56:39.576 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.459035441Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-02-10T05:56:39.576 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.459941529Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-02-10T05:56:39.576 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.46081886Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-02-10T05:56:39.576 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.461496237Z level=info msg="Executing migration" id="create team table" 2024-02-10T05:56:39.576 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.462446267Z level=info msg="Executing migration" id="add index team.org_id" 2024-02-10T05:56:39.576 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.463060123Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-02-10T05:56:39.577 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.463924182Z level=info msg="Executing migration" id="create team member table" 2024-02-10T05:56:39.577 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.46485124Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-02-10T05:56:39.577 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.465666095Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-02-10T05:56:39.577 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.466375291Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-02-10T05:56:39.577 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.467390671Z level=info msg="Executing migration" id="Add column email to team table" 2024-02-10T05:56:39.577 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.468219887Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-02-10T05:56:39.577 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.469082788Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-02-10T05:56:39.577 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.469946194Z level=info msg="Executing migration" id="create dashboard acl table" 2024-02-10T05:56:39.577 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.470814896Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-02-10T05:56:39.577 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.471791189Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-02-10T05:56:39.577 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.472454989Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-02-10T05:56:39.577 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.472916936Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-02-10T05:56:39.577 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.47394717Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-02-10T05:56:39.578 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.474761955Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-02-10T05:56:39.578 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.475453535Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-02-10T05:56:39.578 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.475884095Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-02-10T05:56:39.578 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.476741987Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-02-10T05:56:39.578 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.477691722Z level=info msg="Executing migration" id="create tag table" 2024-02-10T05:56:39.578 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.478390309Z level=info msg="Executing migration" id="add index tag.key_value" 2024-02-10T05:56:39.578 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.478894796Z level=info msg="Executing migration" id="create login attempt table" 2024-02-10T05:56:39.578 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.479718689Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-02-10T05:56:39.578 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.480548768Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-02-10T05:56:39.578 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.481508488Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-02-10T05:56:39.578 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.483482614Z level=info msg="Executing migration" id="create login_attempt v2" 2024-02-10T05:56:39.578 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.486365969Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-02-10T05:56:39.578 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.489100051Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-02-10T05:56:39.579 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.491965964Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-02-10T05:56:39.579 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.494813355Z level=info msg="Executing migration" id="create user auth table" 2024-02-10T05:56:39.579 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.497648266Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-02-10T05:56:39.579 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.500376149Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-02-10T05:56:39.579 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.503120837Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-02-10T05:56:39.579 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.503883949Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-02-10T05:56:39.579 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.50661246Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-02-10T05:56:39.579 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.509493108Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-02-10T05:56:39.579 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.511980964Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-02-10T05:56:39.579 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.512468084Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-02-10T05:56:39.579 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.515271371Z level=info msg="Executing migration" id="create server_lock table" 2024-02-10T05:56:39.579 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.518114972Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-02-10T05:56:39.579 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.521123515Z level=info msg="Executing migration" id="create user auth token table" 2024-02-10T05:56:39.580 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.521627881Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-02-10T05:56:39.580 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.524442318Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-02-10T05:56:39.580 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.527233416Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-02-10T05:56:39.580 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.529810537Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-02-10T05:56:39.580 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.530724618Z level=info msg="Executing migration" id="create cache_data table" 2024-02-10T05:56:39.580 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.533426815Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-02-10T05:56:39.580 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.536121239Z level=info msg="Executing migration" id="create short_url table v1" 2024-02-10T05:56:39.580 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.5391589Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-02-10T05:56:39.580 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.539979319Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-02-10T05:56:39.580 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.542573931Z level=info msg="Executing migration" id="delete alert_definition table" 2024-02-10T05:56:39.580 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.542983635Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-02-10T05:56:39.580 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.543455383Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-02-10T05:56:39.580 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.543932937Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-02-10T05:56:39.580 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.544408744Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-02-10T05:56:39.581 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.544683182Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-02-10T05:56:39.581 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.545142122Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-02-10T05:56:39.581 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.545572784Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-02-10T05:56:39.581 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.546116863Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-02-10T05:56:39.581 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.546585196Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-02-10T05:56:39.581 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.547499328Z level=info msg="Executing migration" id="drop alert_definition table" 2024-02-10T05:56:39.581 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.548317715Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-02-10T05:56:39.581 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.549177371Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-02-10T05:56:39.581 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.549630424Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-02-10T05:56:39.581 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.550429292Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-02-10T05:56:39.581 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.551223991Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-02-10T05:56:39.581 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.552161917Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-02-10T05:56:39.581 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.552732392Z level=info msg="Executing migration" id="create alert_instance table" 2024-02-10T05:56:39.581 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.553515145Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-02-10T05:56:39.582 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.554444669Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-02-10T05:56:39.582 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.555204309Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-02-10T05:56:39.582 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.556047114Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-02-10T05:56:39.582 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.556965319Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-02-10T05:56:39.582 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.557628408Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-02-10T05:56:39.582 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.560985317Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-02-10T05:56:39.582 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.56434335Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-02-10T05:56:39.582 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.565199202Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-02-10T05:56:39.582 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.566112166Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-02-10T05:56:39.582 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.567145218Z level=info msg="Executing migration" id="create alert_rule table" 2024-02-10T05:56:39.582 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.567587807Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-02-10T05:56:39.582 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.568419455Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-02-10T05:56:39.582 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.569209869Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-02-10T05:56:39.583 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.570049805Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-02-10T05:56:39.583 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.570390226Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-02-10T05:56:39.583 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.57139803Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-02-10T05:56:39.797 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:56:39.817 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.572390602Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-02-10T05:56:39.818 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.573382427Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-02-10T05:56:39.818 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.573843173Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-02-10T05:56:39.818 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.574704256Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-02-10T05:56:39.818 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.575750367Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-02-10T05:56:39.818 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.576719677Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-02-10T05:56:39.818 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.577224324Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-02-10T05:56:39.818 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.58011219Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-02-10T05:56:39.818 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.582924795Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-02-10T05:56:39.818 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.585556216Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-02-10T05:56:39.818 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.586017683Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-02-10T05:56:39.818 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.589072743Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-02-10T05:56:39.818 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.591625308Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-02-10T05:56:39.818 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.594446019Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-02-10T05:56:39.819 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.595366686Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-02-10T05:56:39.819 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.59820161Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-02-10T05:56:39.819 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.601015596Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-02-10T05:56:39.819 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.603953118Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-02-10T05:56:39.819 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.604826312Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-02-10T05:56:39.819 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.607748237Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-02-10T05:56:39.819 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.61067838Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-02-10T05:56:39.819 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.613380058Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-02-10T05:56:39.819 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.613720686Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-02-10T05:56:39.819 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.616616861Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-02-10T05:56:39.819 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.619212446Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-02-10T05:56:39.819 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.622017243Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-02-10T05:56:39.819 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.622445124Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-02-10T05:56:39.819 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.625213501Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-02-10T05:56:39.820 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.627829001Z level=info msg="Executing migration" id="create provenance_type table" 2024-02-10T05:56:39.820 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.630626908Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-02-10T05:56:39.820 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.631201896Z level=info msg="Executing migration" id="create alert_image table" 2024-02-10T05:56:39.820 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.633893982Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-02-10T05:56:39.820 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.636637399Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-02-10T05:56:39.820 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.639245909Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-02-10T05:56:39.820 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.639764648Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-02-10T05:56:39.820 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.642541376Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-02-10T05:56:39.821 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.642671369Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-02-10T05:56:39.821 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.64512699Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-02-10T05:56:39.821 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.647912633Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-02-10T05:56:39.821 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.648479344Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-02-10T05:56:39.821 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.648596242Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-02-10T05:56:39.822 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.648704377Z level=info msg="alerts found to migrate" alerts=0 2024-02-10T05:56:39.822 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.651253383Z level=info msg="Executing migration" id="create library_element table v1" 2024-02-10T05:56:39.822 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.653953538Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-02-10T05:56:39.822 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.656259049Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-02-10T05:56:39.822 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.656687042Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-02-10T05:56:39.822 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.657451022Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-02-10T05:56:39.822 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.658346676Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-02-10T05:56:39.822 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.659012001Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-02-10T05:56:39.822 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.659413014Z level=info msg="Executing migration" id="create data_keys table" 2024-02-10T05:56:39.822 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.660289481Z level=info msg="Executing migration" id="create secrets table" 2024-02-10T05:56:39.822 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.661042796Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-02-10T05:56:39.822 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.665311759Z level=info msg="Executing migration" id="add name column into data_keys" 2024-02-10T05:56:39.822 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.666295205Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-02-10T05:56:39.822 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.667213516Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-02-10T05:56:39.822 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.671556867Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-02-10T05:56:39.823 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.675672177Z level=info msg="Executing migration" id="create kv_store table v1" 2024-02-10T05:56:39.823 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.676120774Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-02-10T05:56:39.823 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.676881082Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-02-10T05:56:39.823 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.677725258Z level=info msg="Executing migration" id="create permission table" 2024-02-10T05:56:39.823 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.678502719Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-02-10T05:56:39.823 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.678997802Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-02-10T05:56:39.823 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.679917767Z level=info msg="Executing migration" id="create role table" 2024-02-10T05:56:39.823 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.680657761Z level=info msg="Executing migration" id="add column display_name" 2024-02-10T05:56:39.823 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.68164533Z level=info msg="Executing migration" id="add column group_name" 2024-02-10T05:56:39.823 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.68263658Z level=info msg="Executing migration" id="add index role.org_id" 2024-02-10T05:56:39.823 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.683478085Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-02-10T05:56:39.823 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.684361665Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-02-10T05:56:39.823 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.685343574Z level=info msg="Executing migration" id="create team role table" 2024-02-10T05:56:39.823 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.685963323Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-02-10T05:56:39.823 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.686764907Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-02-10T05:56:39.824 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.6875629Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-02-10T05:56:39.824 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.688560972Z level=info msg="Executing migration" id="create user role table" 2024-02-10T05:56:39.824 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.68900376Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-02-10T05:56:39.824 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.689993967Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-02-10T05:56:39.824 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.690663535Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-02-10T05:56:39.824 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.691668383Z level=info msg="Executing migration" id="create builtin role table" 2024-02-10T05:56:39.824 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.692167323Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-02-10T05:56:39.824 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.693136202Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-02-10T05:56:39.824 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.693848156Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-02-10T05:56:39.825 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.696942405Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-02-10T05:56:39.825 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.697514245Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-02-10T05:56:39.825 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.70026159Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-02-10T05:56:39.825 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.703164202Z level=info msg="Executing migration" id="add unique index role.uid" 2024-02-10T05:56:39.825 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.705962522Z level=info msg="Executing migration" id="create seed assignment table" 2024-02-10T05:56:39.825 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.706428464Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-02-10T05:56:39.825 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.709334356Z level=info msg="Executing migration" id="add column hidden to role table" 2024-02-10T05:56:39.825 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.711973348Z level=info msg="Executing migration" id="create query_history table v1" 2024-02-10T05:56:39.825 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.71460837Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-02-10T05:56:39.825 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.715131872Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-02-10T05:56:39.825 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.718104282Z level=info msg="Executing migration" id="teams permissions migration" 2024-02-10T05:56:39.825 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.720733094Z level=info msg="Executing migration" id="dashboard permissions" 2024-02-10T05:56:39.825 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.723525916Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-02-10T05:56:39.825 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.72395524Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-02-10T05:56:39.826 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.726949226Z level=info msg="Executing migration" id="alerting notification permissions" 2024-02-10T05:56:39.826 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.729659376Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-02-10T05:56:39.826 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.732223154Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-02-10T05:56:39.826 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.732699315Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-02-10T05:56:39.826 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.735442228Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-02-10T05:56:39.826 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.738139352Z level=info msg="Executing migration" id="create correlation table v1" 2024-02-10T05:56:39.826 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.740680391Z level=info msg="Executing migration" id="add index correlations.uid" 2024-02-10T05:56:39.826 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.741148612Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-02-10T05:56:39.826 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.743970339Z level=info msg="Executing migration" id="add correlation config column" 2024-02-10T05:56:39.826 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.746693662Z level=info msg="Executing migration" id="create entity_events table" 2024-02-10T05:56:39.826 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.749146379Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-02-10T05:56:39.826 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.749617918Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-02-10T05:56:39.826 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.749709615Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-02-10T05:56:39.826 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.749998304Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-10T05:56:39.826 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.750085812Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-10T05:56:39.827 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.75037265Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-02-10T05:56:39.827 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.750738262Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-02-10T05:56:39.827 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.751165878Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-02-10T05:56:39.827 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.751637079Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-10T05:56:39.827 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.752093798Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-02-10T05:56:39.827 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.752549421Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-02-10T05:56:39.827 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.752959554Z level=info msg="Executing migration" id="Drop public config table" 2024-02-10T05:56:39.827 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.753377801Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-02-10T05:56:39.827 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.753776146Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-02-10T05:56:39.827 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.75422755Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-02-10T05:56:39.827 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.754698929Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-02-10T05:56:39.827 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.757570564Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-02-10T05:56:39.827 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.760557057Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-02-10T05:56:39.827 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.761610734Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-02-10T05:56:39.828 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.762640837Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-02-10T05:56:39.828 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.762994529Z level=info msg="Executing migration" id="add share column" 2024-02-10T05:56:39.828 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.764013772Z level=info msg="Executing migration" id="create default alerting folders" 2024-02-10T05:56:39.828 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.764499735Z level=info msg="Executing migration" id="create file table" 2024-02-10T05:56:39.828 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.764963957Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-02-10T05:56:39.828 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.767725359Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-02-10T05:56:39.828 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.768298361Z level=info msg="Executing migration" id="create file_meta table" 2024-02-10T05:56:39.828 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.768729396Z level=info msg="Executing migration" id="file table idx: path key" 2024-02-10T05:56:39.828 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.771432897Z level=info msg="Executing migration" id="set path collation in file table" 2024-02-10T05:56:39.828 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.774185021Z level=info msg="Executing migration" id="managed permissions migration" 2024-02-10T05:56:39.828 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.776890297Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-02-10T05:56:39.828 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.777284994Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-02-10T05:56:39.828 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.779863789Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-02-10T05:56:39.828 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.780940915Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-02-10T05:56:39.829 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.782722373Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-02-10T05:56:39.829 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.785328596Z level=info msg="Executing migration" id="update group index for alert rules" 2024-02-10T05:56:39.829 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.785950007Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-02-10T05:56:39.829 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.788728362Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-02-10T05:56:39.829 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.791407267Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-02-10T05:56:39.829 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.793919514Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-02-10T05:56:39.829 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.794948158Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-02-10T05:56:39.829 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.797651114Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-02-10T05:56:39.829 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.807432012Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-02-10T05:56:39.829 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.810104922Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-02-10T05:56:39.829 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.810597047Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-02-10T05:56:39.829 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.81399182Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-02-10T05:56:39.829 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.814841497Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-02-10T05:56:39.829 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.815548478Z level=info msg="Executing migration" id="create folder table" 2024-02-10T05:56:39.830 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.815961178Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-02-10T05:56:39.830 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.816426633Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-02-10T05:56:39.830 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.816904143Z level=info msg="Executing migration" id="Update folder title length" 2024-02-10T05:56:39.830 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.817798096Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-02-10T05:56:39.830 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=migrator t=2024-02-10T05:56:39.818552344Z level=info msg="migrations completed" performed=485 skipped=0 duration=679.839954ms 2024-02-10T05:56:39.830 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=sqlstore t=2024-02-10T05:56:39.818968044Z level=info msg="Created default organization" 2024-02-10T05:56:40.119 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=plugin.loader t=2024-02-10T05:56:39.874590333Z level=info msg="Plugin registered" pluginID=input 2024-02-10T05:56:40.120 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=plugin.loader t=2024-02-10T05:56:39.901427694Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-02-10T05:56:40.120 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=plugin.loader t=2024-02-10T05:56:39.901446787Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-02-10T05:56:40.120 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=secrets t=2024-02-10T05:56:39.901478503Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-02-10T05:56:40.120 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=query_data t=2024-02-10T05:56:39.903585588Z level=info msg="Query Service initialization" 2024-02-10T05:56:40.120 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=live.push_http t=2024-02-10T05:56:39.905988186Z level=info msg="Live Push Gateway initialization" 2024-02-10T05:56:40.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:40 smithi064 ceph-mon[25185]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:56:40.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:40 smithi064 ceph-mon[25185]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:40.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:40 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:40.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:40 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:40.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:40 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:40.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:40 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:40.175 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:40 smithi064 ceph-mon[25185]: Deploying daemon node-exporter.a on smithi064 2024-02-10T05:56:40.176 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:40 smithi064 ceph-mon[38958]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:56:40.176 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:40 smithi064 ceph-mon[38958]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:40.176 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:40 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:40.176 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:40 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:40.176 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:40 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:40.176 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:40 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:40.176 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:40 smithi064 ceph-mon[38958]: Deploying daemon node-exporter.a on smithi064 2024-02-10T05:56:40.395 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:40 smithi176 ceph-mon[30294]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:56:40.396 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:40 smithi176 ceph-mon[30294]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:40.396 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:40 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:40.396 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:40 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:40.396 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:40 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:40.396 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:40 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:40.396 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:40 smithi176 ceph-mon[30294]: Deploying daemon node-exporter.a on smithi064 2024-02-10T05:56:40.738 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:40 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=infra.usagestats.collector t=2024-02-10T05:56:40.586189761Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-02-10T05:56:40.739 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:40 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=provisioning.datasources t=2024-02-10T05:56:40.587396245Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-10T05:56:40.739 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:40 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=provisioning.datasources t=2024-02-10T05:56:40.598501261Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-02-10T05:56:40.739 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:40 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=provisioning.alerting t=2024-02-10T05:56:40.609682201Z level=info msg="starting to provision alerting" 2024-02-10T05:56:40.739 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:40 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=provisioning.alerting t=2024-02-10T05:56:40.609721304Z level=info msg="finished to provision alerting" 2024-02-10T05:56:40.739 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:40 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=grafanaStorageLogger t=2024-02-10T05:56:40.609878701Z level=info msg="storage starting" 2024-02-10T05:56:40.739 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:40 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=ngalert.state.manager t=2024-02-10T05:56:40.610108596Z level=info msg="Warming state cache for startup" 2024-02-10T05:56:40.739 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:40 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=ngalert.state.manager t=2024-02-10T05:56:40.610228108Z level=info msg="State cache has been initialized" states=0 duration=115.98µs 2024-02-10T05:56:40.739 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:40 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=ticker t=2024-02-10T05:56:40.610291656Z level=info msg=starting first_tick=2024-02-10T05:56:50Z 2024-02-10T05:56:40.739 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:40 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=ngalert.multiorg.alertmanager t=2024-02-10T05:56:40.610314898Z level=info msg="starting MultiOrg Alertmanager" 2024-02-10T05:56:40.739 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:40 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=http.server t=2024-02-10T05:56:40.61218479Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-02-10T05:56:40.739 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:56:40 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a[69729]: logger=sqlstore.transactions t=2024-02-10T05:56:40.628035493Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-02-10T05:56:41.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:41 smithi064 ceph-mon[25185]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:41.345 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:41 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:41.345 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:41 smithi064 ceph-mon[38958]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:41.345 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:41 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:41.345 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:41 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: ::ffff:172.21.15.176 - - [10/Feb/2024:05:56:41] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T05:56:41.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:41 smithi176 ceph-mon[30294]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:41.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:41 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:41.798 INFO:teuthology.orchestra.run.smithi064.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-02-10T05:56:41.798 INFO:teuthology.orchestra.run.smithi064.stdout:smithi064 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT534400C2400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-02-10T05:56:41.798 INFO:teuthology.orchestra.run.smithi064.stdout:smithi064 /dev/nvme1n1 ssd Linux_96fb1ebcbe27fbb0f429 89.4G No 3m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-02-10T05:56:41.798 INFO:teuthology.orchestra.run.smithi064.stdout:smithi064 /dev/nvme2n1 ssd Linux_84314c54f9fa6806e2d9 89.4G No 3m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-02-10T05:56:41.798 INFO:teuthology.orchestra.run.smithi064.stdout:smithi064 /dev/nvme3n1 ssd Linux_476a0469b662288393e0 89.4G No 3m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-02-10T05:56:41.798 INFO:teuthology.orchestra.run.smithi064.stdout:smithi064 /dev/nvme4n1 ssd Linux_0368b02efd5ab14ae36a 89.4G No 3m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-02-10T05:56:41.798 INFO:teuthology.orchestra.run.smithi064.stdout:smithi176 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT62330099400BGN 372G No 114s ago Has a FileSystem, LVM detected 2024-02-10T05:56:41.798 INFO:teuthology.orchestra.run.smithi064.stdout:smithi176 /dev/nvme1n1 ssd Linux_d5f950076274d28fd035 89.4G No 114s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-02-10T05:56:41.798 INFO:teuthology.orchestra.run.smithi064.stdout:smithi176 /dev/nvme2n1 ssd Linux_f55c4e9ca67a7fbf5c07 89.4G No 114s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-02-10T05:56:41.799 INFO:teuthology.orchestra.run.smithi064.stdout:smithi176 /dev/nvme3n1 ssd Linux_766529302a64b7756be0 89.4G No 114s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-02-10T05:56:41.799 INFO:teuthology.orchestra.run.smithi064.stdout:smithi176 /dev/nvme4n1 ssd Linux_359591fff069553d4dab 89.4G No 114s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-02-10T05:56:42.798 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-02-10T05:56:43.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:43 smithi176 ceph-mon[30294]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:43.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:43 smithi176 ceph-mon[30294]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:43.613 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:56:43.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:43 smithi064 ceph-mon[25185]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:43.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:43 smithi064 ceph-mon[25185]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:43.626 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:43 smithi064 ceph-mon[38958]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:43.626 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:43 smithi064 ceph-mon[38958]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:44.976 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 podman[84055]: 2024-02-10 05:56:44.976527061 +0000 UTC m=+3.834964314 container init fb307795377db7435354bf8f87adbc58022489e20ff007f0081408a95ca52fe3 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T05:56:45.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:45 smithi064 ceph-mon[25185]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:45.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:45 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:45.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:45 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:45.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:45 smithi064 ceph-mon[38958]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:45.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:45 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:45.377 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:45 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:45.377 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 podman[84055]: 2024-02-10 05:56:44.979506355 +0000 UTC m=+3.837943616 container start fb307795377db7435354bf8f87adbc58022489e20ff007f0081408a95ca52fe3 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T05:56:45.378 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.983Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-02-10T05:56:45.378 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.983Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-02-10T05:56:45.378 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.984Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-02-10T05:56:45.378 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.984Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-02-10T05:56:45.378 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.984Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-02-10T05:56:45.378 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.984Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-10T05:56:45.378 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.984Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-02-10T05:56:45.378 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.984Z caller=node_exporter.go:117 level=info collector=arp 2024-02-10T05:56:45.378 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.984Z caller=node_exporter.go:117 level=info collector=bcache 2024-02-10T05:56:45.378 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.984Z caller=node_exporter.go:117 level=info collector=bonding 2024-02-10T05:56:45.378 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.984Z caller=node_exporter.go:117 level=info collector=btrfs 2024-02-10T05:56:45.378 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.984Z caller=node_exporter.go:117 level=info collector=conntrack 2024-02-10T05:56:45.379 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=cpu 2024-02-10T05:56:45.379 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-02-10T05:56:45.379 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=diskstats 2024-02-10T05:56:45.379 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=dmi 2024-02-10T05:56:45.379 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=edac 2024-02-10T05:56:45.379 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=entropy 2024-02-10T05:56:45.379 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-02-10T05:56:45.379 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=filefd 2024-02-10T05:56:45.379 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=filesystem 2024-02-10T05:56:45.379 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=hwmon 2024-02-10T05:56:45.379 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=infiniband 2024-02-10T05:56:45.379 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=ipvs 2024-02-10T05:56:45.380 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=loadavg 2024-02-10T05:56:45.380 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=mdadm 2024-02-10T05:56:45.380 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=meminfo 2024-02-10T05:56:45.380 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=netclass 2024-02-10T05:56:45.380 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=netdev 2024-02-10T05:56:45.380 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=netstat 2024-02-10T05:56:45.380 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=nfs 2024-02-10T05:56:45.380 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=nfsd 2024-02-10T05:56:45.380 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=nvme 2024-02-10T05:56:45.380 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=os 2024-02-10T05:56:45.380 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-02-10T05:56:45.380 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=pressure 2024-02-10T05:56:45.380 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=rapl 2024-02-10T05:56:45.381 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=schedstat 2024-02-10T05:56:45.381 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=selinux 2024-02-10T05:56:45.381 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=sockstat 2024-02-10T05:56:45.381 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=softnet 2024-02-10T05:56:45.381 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=stat 2024-02-10T05:56:45.381 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=tapestats 2024-02-10T05:56:45.381 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=textfile 2024-02-10T05:56:45.381 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-02-10T05:56:45.381 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=time 2024-02-10T05:56:45.381 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-02-10T05:56:45.381 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=uname 2024-02-10T05:56:45.381 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=vmstat 2024-02-10T05:56:45.382 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=xfs 2024-02-10T05:56:45.382 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=node_exporter.go:117 level=info collector=zfs 2024-02-10T05:56:45.382 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-02-10T05:56:45.382 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:44 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a[84239]: ts=2024-02-10T05:56:44.985Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-02-10T05:56:45.382 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:45 smithi064 bash[84055]: fb307795377db7435354bf8f87adbc58022489e20ff007f0081408a95ca52fe3 2024-02-10T05:56:45.382 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:56:45 smithi064 systemd[1]: Started Ceph node-exporter.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:56:45.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:45 smithi176 ceph-mon[30294]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:45.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:45 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:45.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:45 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:46.169 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:46 smithi176 systemd[1]: Starting Ceph node-exporter.b for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:56:46.346 INFO:teuthology.orchestra.run.smithi064.stdout:service_type: alertmanager 2024-02-10T05:56:46.347 INFO:teuthology.orchestra.run.smithi064.stdout:service_name: alertmanager 2024-02-10T05:56:46.347 INFO:teuthology.orchestra.run.smithi064.stdout:placement: 2024-02-10T05:56:46.347 INFO:teuthology.orchestra.run.smithi064.stdout: count: 1 2024-02-10T05:56:46.347 INFO:teuthology.orchestra.run.smithi064.stdout: hosts: 2024-02-10T05:56:46.347 INFO:teuthology.orchestra.run.smithi064.stdout: - smithi064=a 2024-02-10T05:56:46.347 INFO:teuthology.orchestra.run.smithi064.stdout:status: 2024-02-10T05:56:46.347 INFO:teuthology.orchestra.run.smithi064.stdout: created: '2024-02-10T05:55:14.293264Z' 2024-02-10T05:56:46.347 INFO:teuthology.orchestra.run.smithi064.stdout: ports: 2024-02-10T05:56:46.347 INFO:teuthology.orchestra.run.smithi064.stdout: - 9093 2024-02-10T05:56:46.347 INFO:teuthology.orchestra.run.smithi064.stdout: - 9094 2024-02-10T05:56:46.347 INFO:teuthology.orchestra.run.smithi064.stdout: running: 0 2024-02-10T05:56:46.347 INFO:teuthology.orchestra.run.smithi064.stdout: size: 1 2024-02-10T05:56:46.347 INFO:teuthology.orchestra.run.smithi064.stdout:events: 2024-02-10T05:56:46.348 INFO:teuthology.orchestra.run.smithi064.stdout:- 2024-02-10T05:56:24.153855Z service:alertmanager [INFO] "service was created" 2024-02-10T05:56:46.348 INFO:teuthology.orchestra.run.smithi064.stdout:--- 2024-02-10T05:56:46.348 INFO:teuthology.orchestra.run.smithi064.stdout:service_type: grafana 2024-02-10T05:56:46.348 INFO:teuthology.orchestra.run.smithi064.stdout:service_name: grafana 2024-02-10T05:56:46.348 INFO:teuthology.orchestra.run.smithi064.stdout:placement: 2024-02-10T05:56:46.348 INFO:teuthology.orchestra.run.smithi064.stdout: count: 1 2024-02-10T05:56:46.348 INFO:teuthology.orchestra.run.smithi064.stdout: hosts: 2024-02-10T05:56:46.348 INFO:teuthology.orchestra.run.smithi064.stdout: - smithi176=a 2024-02-10T05:56:46.348 INFO:teuthology.orchestra.run.smithi064.stdout:spec: 2024-02-10T05:56:46.348 INFO:teuthology.orchestra.run.smithi064.stdout: anonymous_access: true 2024-02-10T05:56:46.348 INFO:teuthology.orchestra.run.smithi064.stdout: protocol: https 2024-02-10T05:56:46.348 INFO:teuthology.orchestra.run.smithi064.stdout:status: 2024-02-10T05:56:46.348 INFO:teuthology.orchestra.run.smithi064.stdout: created: '2024-02-10T05:55:17.883743Z' 2024-02-10T05:56:46.349 INFO:teuthology.orchestra.run.smithi064.stdout: ports: 2024-02-10T05:56:46.349 INFO:teuthology.orchestra.run.smithi064.stdout: - 3000 2024-02-10T05:56:46.349 INFO:teuthology.orchestra.run.smithi064.stdout: running: 0 2024-02-10T05:56:46.349 INFO:teuthology.orchestra.run.smithi064.stdout: size: 1 2024-02-10T05:56:46.349 INFO:teuthology.orchestra.run.smithi064.stdout:events: 2024-02-10T05:56:46.349 INFO:teuthology.orchestra.run.smithi064.stdout:- 2024-02-10T05:56:39.155851Z service:grafana [INFO] "service was created" 2024-02-10T05:56:46.349 INFO:teuthology.orchestra.run.smithi064.stdout:--- 2024-02-10T05:56:46.349 INFO:teuthology.orchestra.run.smithi064.stdout:service_type: iscsi 2024-02-10T05:56:46.349 INFO:teuthology.orchestra.run.smithi064.stdout:service_id: datapool 2024-02-10T05:56:46.349 INFO:teuthology.orchestra.run.smithi064.stdout:service_name: iscsi.datapool 2024-02-10T05:56:46.349 INFO:teuthology.orchestra.run.smithi064.stdout:placement: 2024-02-10T05:56:46.349 INFO:teuthology.orchestra.run.smithi064.stdout: count: 1 2024-02-10T05:56:46.349 INFO:teuthology.orchestra.run.smithi064.stdout: hosts: 2024-02-10T05:56:46.350 INFO:teuthology.orchestra.run.smithi064.stdout: - smithi176=iscsi.a 2024-02-10T05:56:46.350 INFO:teuthology.orchestra.run.smithi064.stdout:spec: 2024-02-10T05:56:46.350 INFO:teuthology.orchestra.run.smithi064.stdout: api_password: admin 2024-02-10T05:56:46.350 INFO:teuthology.orchestra.run.smithi064.stdout: api_port: 5000 2024-02-10T05:56:46.350 INFO:teuthology.orchestra.run.smithi064.stdout: api_user: admin 2024-02-10T05:56:46.350 INFO:teuthology.orchestra.run.smithi064.stdout: pool: datapool 2024-02-10T05:56:46.350 INFO:teuthology.orchestra.run.smithi064.stdout: trusted_ip_list: 172.21.15.176 2024-02-10T05:56:46.350 INFO:teuthology.orchestra.run.smithi064.stdout:status: 2024-02-10T05:56:46.350 INFO:teuthology.orchestra.run.smithi064.stdout: created: '2024-02-10T05:55:03.400185Z' 2024-02-10T05:56:46.350 INFO:teuthology.orchestra.run.smithi064.stdout: last_refresh: '2024-02-10T05:55:39.151292Z' 2024-02-10T05:56:46.350 INFO:teuthology.orchestra.run.smithi064.stdout: ports: 2024-02-10T05:56:46.350 INFO:teuthology.orchestra.run.smithi064.stdout: - 5000 2024-02-10T05:56:46.350 INFO:teuthology.orchestra.run.smithi064.stdout: running: 1 2024-02-10T05:56:46.350 INFO:teuthology.orchestra.run.smithi064.stdout: size: 1 2024-02-10T05:56:46.350 INFO:teuthology.orchestra.run.smithi064.stdout:--- 2024-02-10T05:56:46.351 INFO:teuthology.orchestra.run.smithi064.stdout:service_type: mgr 2024-02-10T05:56:46.351 INFO:teuthology.orchestra.run.smithi064.stdout:service_name: mgr 2024-02-10T05:56:46.351 INFO:teuthology.orchestra.run.smithi064.stdout:placement: 2024-02-10T05:56:46.351 INFO:teuthology.orchestra.run.smithi064.stdout: count: 2 2024-02-10T05:56:46.351 INFO:teuthology.orchestra.run.smithi064.stdout: hosts: 2024-02-10T05:56:46.351 INFO:teuthology.orchestra.run.smithi064.stdout: - smithi064=y 2024-02-10T05:56:46.351 INFO:teuthology.orchestra.run.smithi064.stdout: - smithi176=x 2024-02-10T05:56:46.351 INFO:teuthology.orchestra.run.smithi064.stdout:status: 2024-02-10T05:56:46.351 INFO:teuthology.orchestra.run.smithi064.stdout: created: '2024-02-10T05:51:17.903462Z' 2024-02-10T05:56:46.351 INFO:teuthology.orchestra.run.smithi064.stdout: last_refresh: '2024-02-10T05:55:39.150862Z' 2024-02-10T05:56:46.351 INFO:teuthology.orchestra.run.smithi064.stdout: running: 2 2024-02-10T05:56:46.351 INFO:teuthology.orchestra.run.smithi064.stdout: size: 2 2024-02-10T05:56:46.351 INFO:teuthology.orchestra.run.smithi064.stdout:--- 2024-02-10T05:56:46.351 INFO:teuthology.orchestra.run.smithi064.stdout:service_type: mon 2024-02-10T05:56:46.352 INFO:teuthology.orchestra.run.smithi064.stdout:service_name: mon 2024-02-10T05:56:46.352 INFO:teuthology.orchestra.run.smithi064.stdout:placement: 2024-02-10T05:56:46.352 INFO:teuthology.orchestra.run.smithi064.stdout: count: 3 2024-02-10T05:56:46.352 INFO:teuthology.orchestra.run.smithi064.stdout: hosts: 2024-02-10T05:56:46.352 INFO:teuthology.orchestra.run.smithi064.stdout: - smithi064:172.21.15.64=a 2024-02-10T05:56:46.352 INFO:teuthology.orchestra.run.smithi064.stdout: - smithi064:[v2:172.21.15.64:3301,v1:172.21.15.64:6790]=c 2024-02-10T05:56:46.352 INFO:teuthology.orchestra.run.smithi064.stdout: - smithi176:172.21.15.176=b 2024-02-10T05:56:46.352 INFO:teuthology.orchestra.run.smithi064.stdout:status: 2024-02-10T05:56:46.352 INFO:teuthology.orchestra.run.smithi064.stdout: created: '2024-02-10T05:50:53.379514Z' 2024-02-10T05:56:46.352 INFO:teuthology.orchestra.run.smithi064.stdout: last_refresh: '2024-02-10T05:55:39.150727Z' 2024-02-10T05:56:46.352 INFO:teuthology.orchestra.run.smithi064.stdout: running: 3 2024-02-10T05:56:46.352 INFO:teuthology.orchestra.run.smithi064.stdout: size: 3 2024-02-10T05:56:46.352 INFO:teuthology.orchestra.run.smithi064.stdout:--- 2024-02-10T05:56:46.352 INFO:teuthology.orchestra.run.smithi064.stdout:service_type: node-exporter 2024-02-10T05:56:46.353 INFO:teuthology.orchestra.run.smithi064.stdout:service_name: node-exporter 2024-02-10T05:56:46.353 INFO:teuthology.orchestra.run.smithi064.stdout:placement: 2024-02-10T05:56:46.353 INFO:teuthology.orchestra.run.smithi064.stdout: count: 2 2024-02-10T05:56:46.353 INFO:teuthology.orchestra.run.smithi064.stdout: hosts: 2024-02-10T05:56:46.353 INFO:teuthology.orchestra.run.smithi064.stdout: - smithi064=a 2024-02-10T05:56:46.353 INFO:teuthology.orchestra.run.smithi064.stdout: - smithi176=b 2024-02-10T05:56:46.353 INFO:teuthology.orchestra.run.smithi064.stdout:status: 2024-02-10T05:56:46.353 INFO:teuthology.orchestra.run.smithi064.stdout: created: '2024-02-10T05:55:10.508734Z' 2024-02-10T05:56:46.353 INFO:teuthology.orchestra.run.smithi064.stdout: ports: 2024-02-10T05:56:46.353 INFO:teuthology.orchestra.run.smithi064.stdout: - 9100 2024-02-10T05:56:46.353 INFO:teuthology.orchestra.run.smithi064.stdout: running: 0 2024-02-10T05:56:46.353 INFO:teuthology.orchestra.run.smithi064.stdout: size: 2 2024-02-10T05:56:46.353 INFO:teuthology.orchestra.run.smithi064.stdout:events: 2024-02-10T05:56:46.353 INFO:teuthology.orchestra.run.smithi064.stdout:- 2024-02-10T05:56:45.185444Z service:node-exporter [INFO] "service was created" 2024-02-10T05:56:46.353 INFO:teuthology.orchestra.run.smithi064.stdout:--- 2024-02-10T05:56:46.354 INFO:teuthology.orchestra.run.smithi064.stdout:service_type: osd 2024-02-10T05:56:46.354 INFO:teuthology.orchestra.run.smithi064.stdout:service_name: osd 2024-02-10T05:56:46.354 INFO:teuthology.orchestra.run.smithi064.stdout:unmanaged: true 2024-02-10T05:56:46.354 INFO:teuthology.orchestra.run.smithi064.stdout:spec: 2024-02-10T05:56:46.354 INFO:teuthology.orchestra.run.smithi064.stdout: filter_logic: AND 2024-02-10T05:56:46.354 INFO:teuthology.orchestra.run.smithi064.stdout: objectstore: bluestore 2024-02-10T05:56:46.354 INFO:teuthology.orchestra.run.smithi064.stdout:status: 2024-02-10T05:56:46.354 INFO:teuthology.orchestra.run.smithi064.stdout: container_image_id: c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 2024-02-10T05:56:46.354 INFO:teuthology.orchestra.run.smithi064.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:56:46.354 INFO:teuthology.orchestra.run.smithi064.stdout: last_refresh: '2024-02-10T05:55:39.150956Z' 2024-02-10T05:56:46.354 INFO:teuthology.orchestra.run.smithi064.stdout: running: 8 2024-02-10T05:56:46.354 INFO:teuthology.orchestra.run.smithi064.stdout: size: 8 2024-02-10T05:56:46.354 INFO:teuthology.orchestra.run.smithi064.stdout:--- 2024-02-10T05:56:46.354 INFO:teuthology.orchestra.run.smithi064.stdout:service_type: prometheus 2024-02-10T05:56:46.355 INFO:teuthology.orchestra.run.smithi064.stdout:service_name: prometheus 2024-02-10T05:56:46.355 INFO:teuthology.orchestra.run.smithi064.stdout:placement: 2024-02-10T05:56:46.355 INFO:teuthology.orchestra.run.smithi064.stdout: count: 1 2024-02-10T05:56:46.355 INFO:teuthology.orchestra.run.smithi064.stdout: hosts: 2024-02-10T05:56:46.355 INFO:teuthology.orchestra.run.smithi064.stdout: - smithi176=a 2024-02-10T05:56:46.355 INFO:teuthology.orchestra.run.smithi064.stdout:status: 2024-02-10T05:56:46.355 INFO:teuthology.orchestra.run.smithi064.stdout: created: '2024-02-10T05:55:07.085682Z' 2024-02-10T05:56:46.355 INFO:teuthology.orchestra.run.smithi064.stdout: last_refresh: '2024-02-10T05:55:39.151375Z' 2024-02-10T05:56:46.355 INFO:teuthology.orchestra.run.smithi064.stdout: ports: 2024-02-10T05:56:46.355 INFO:teuthology.orchestra.run.smithi064.stdout: - 9095 2024-02-10T05:56:46.355 INFO:teuthology.orchestra.run.smithi064.stdout: running: 1 2024-02-10T05:56:46.355 INFO:teuthology.orchestra.run.smithi064.stdout: size: 1 2024-02-10T05:56:46.355 INFO:teuthology.orchestra.run.smithi064.stdout:--- 2024-02-10T05:56:46.355 INFO:teuthology.orchestra.run.smithi064.stdout:service_type: rgw 2024-02-10T05:56:46.355 INFO:teuthology.orchestra.run.smithi064.stdout:service_id: foo.a 2024-02-10T05:56:46.356 INFO:teuthology.orchestra.run.smithi064.stdout:service_name: rgw.foo.a 2024-02-10T05:56:46.356 INFO:teuthology.orchestra.run.smithi064.stdout:placement: 2024-02-10T05:56:46.356 INFO:teuthology.orchestra.run.smithi064.stdout: count: 1 2024-02-10T05:56:46.356 INFO:teuthology.orchestra.run.smithi064.stdout: hosts: 2024-02-10T05:56:46.356 INFO:teuthology.orchestra.run.smithi064.stdout: - smithi064=foo.a 2024-02-10T05:56:46.356 INFO:teuthology.orchestra.run.smithi064.stdout:status: 2024-02-10T05:56:46.356 INFO:teuthology.orchestra.run.smithi064.stdout: created: '2024-02-10T05:54:55.580657Z' 2024-02-10T05:56:46.356 INFO:teuthology.orchestra.run.smithi064.stdout: last_refresh: '2024-02-10T05:56:10.489094Z' 2024-02-10T05:56:46.356 INFO:teuthology.orchestra.run.smithi064.stdout: ports: 2024-02-10T05:56:46.356 INFO:teuthology.orchestra.run.smithi064.stdout: - 80 2024-02-10T05:56:46.356 INFO:teuthology.orchestra.run.smithi064.stdout: running: 1 2024-02-10T05:56:46.356 INFO:teuthology.orchestra.run.smithi064.stdout: size: 1 2024-02-10T05:56:46.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:46 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:46.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:46 smithi176 ceph-mon[30294]: Deploying daemon node-exporter.b on smithi176 2024-02-10T05:56:46.489 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:46 smithi176 bash[69990]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-02-10T05:56:46.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:46 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:46.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:46 smithi064 ceph-mon[25185]: Deploying daemon node-exporter.b on smithi176 2024-02-10T05:56:46.626 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:46 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:46.626 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:46 smithi064 ceph-mon[38958]: Deploying daemon node-exporter.b on smithi176 2024-02-10T05:56:47.093 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-02-10T05:56:47.185 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:46 smithi176 bash[69990]: Getting image source signatures 2024-02-10T05:56:47.185 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:46 smithi176 bash[69990]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-02-10T05:56:47.185 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:46 smithi176 bash[69990]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-02-10T05:56:47.186 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:46 smithi176 bash[69990]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-02-10T05:56:47.344 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:56:47.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:47 smithi064 ceph-mon[25185]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:47.376 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:47 smithi064 ceph-mon[25185]: from='client.24641 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-02-10T05:56:47.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:47 smithi064 ceph-mon[38958]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:47.376 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:47 smithi064 ceph-mon[38958]: from='client.24641 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-02-10T05:56:47.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:47 smithi176 ceph-mon[30294]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-10T05:56:47.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:47 smithi176 ceph-mon[30294]: from='client.24641 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-02-10T05:56:48.391 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 bash[69990]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-02-10T05:56:48.392 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 bash[69990]: Writing manifest to image destination 2024-02-10T05:56:48.696 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[66249]: debug there is no tcmu-runner data available 2024-02-10T05:56:48.696 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 podman[69990]: 2024-02-10 05:56:48.352286181 +0000 UTC m=+2.083196414 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-02-10T05:56:48.697 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 podman[69990]: 2024-02-10 05:56:48.461506459 +0000 UTC m=+2.192416685 container create 135900a9fea48ab45ddb6c2dcbffc79e84a1a9fb33ad9c98ac96db37481f9394 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T05:56:48.988 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 podman[69990]: 2024-02-10 05:56:48.696956627 +0000 UTC m=+2.427866853 container init 135900a9fea48ab45ddb6c2dcbffc79e84a1a9fb33ad9c98ac96db37481f9394 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T05:56:48.989 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 podman[69990]: 2024-02-10 05:56:48.6997367 +0000 UTC m=+2.430646929 container start 135900a9fea48ab45ddb6c2dcbffc79e84a1a9fb33ad9c98ac96db37481f9394 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T05:56:48.989 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.703Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-02-10T05:56:48.989 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.703Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-02-10T05:56:48.989 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.703Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-02-10T05:56:48.989 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.703Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-10T05:56:48.989 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.703Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-02-10T05:56:48.989 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.703Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-02-10T05:56:48.989 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-02-10T05:56:48.989 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=arp 2024-02-10T05:56:48.989 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=bcache 2024-02-10T05:56:48.989 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=bonding 2024-02-10T05:56:48.990 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=btrfs 2024-02-10T05:56:48.990 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=conntrack 2024-02-10T05:56:48.990 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=cpu 2024-02-10T05:56:48.990 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-02-10T05:56:48.990 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=diskstats 2024-02-10T05:56:48.990 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=dmi 2024-02-10T05:56:48.990 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=edac 2024-02-10T05:56:48.991 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=entropy 2024-02-10T05:56:48.991 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-02-10T05:56:48.991 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=filefd 2024-02-10T05:56:48.991 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=filesystem 2024-02-10T05:56:48.991 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=hwmon 2024-02-10T05:56:48.991 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=infiniband 2024-02-10T05:56:48.991 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=ipvs 2024-02-10T05:56:48.991 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=loadavg 2024-02-10T05:56:48.991 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=mdadm 2024-02-10T05:56:48.991 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=meminfo 2024-02-10T05:56:48.991 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=netclass 2024-02-10T05:56:48.991 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=netdev 2024-02-10T05:56:48.992 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=netstat 2024-02-10T05:56:48.992 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=nfs 2024-02-10T05:56:48.992 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=nfsd 2024-02-10T05:56:48.992 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=nvme 2024-02-10T05:56:48.992 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=os 2024-02-10T05:56:48.992 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-02-10T05:56:48.992 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=pressure 2024-02-10T05:56:48.992 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=rapl 2024-02-10T05:56:48.992 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=schedstat 2024-02-10T05:56:48.992 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=selinux 2024-02-10T05:56:48.992 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=sockstat 2024-02-10T05:56:48.992 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=softnet 2024-02-10T05:56:48.992 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=stat 2024-02-10T05:56:48.992 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=tapestats 2024-02-10T05:56:48.993 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=textfile 2024-02-10T05:56:48.993 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-02-10T05:56:48.993 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=time 2024-02-10T05:56:48.993 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-02-10T05:56:48.993 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=uname 2024-02-10T05:56:48.993 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.704Z caller=node_exporter.go:117 level=info collector=vmstat 2024-02-10T05:56:48.993 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.705Z caller=node_exporter.go:117 level=info collector=xfs 2024-02-10T05:56:48.993 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.705Z caller=node_exporter.go:117 level=info collector=zfs 2024-02-10T05:56:48.993 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.705Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-02-10T05:56:48.993 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b[70040]: ts=2024-02-10T05:56:48.705Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-02-10T05:56:48.993 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 bash[69990]: 135900a9fea48ab45ddb6c2dcbffc79e84a1a9fb33ad9c98ac96db37481f9394 2024-02-10T05:56:48.993 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:56:48 smithi176 systemd[1]: Started Ceph node-exporter.b for 00e86dbe-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:56:49.254 INFO:teuthology.orchestra.run.smithi064.stdout:osd 8 70s ago - 2024-02-10T05:56:49.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:49 smithi176 ceph-mon[30294]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:49.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:49 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:49.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:49 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:49.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:49 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:49.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:49 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:49.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:49 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:49.489 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:49 smithi176 ceph-mon[30294]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:49.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:49 smithi064 ceph-mon[38958]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:49.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:49 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:49.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:49 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:49.544 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:49 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:49.545 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:49 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:49.545 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:49 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:49.545 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:49 smithi064 ceph-mon[38958]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:49.545 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:49 smithi064 ceph-mon[25185]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-10T05:56:49.545 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:49 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:49.545 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:49 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:49.545 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:49 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:49.545 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:49 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:49.545 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:49 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' 2024-02-10T05:56:49.545 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:49 smithi064 ceph-mon[25185]: from='mgr.14556 172.21.15.64:0/4062243627' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:56:49.914 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-10T05:56:49.923 INFO:tasks.cephadm:Teardown begin 2024-02-10T05:56:49.924 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:56:49.953 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:56:49.982 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-10T05:56:49.982 DEBUG:teuthology.orchestra.run.smithi064:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T05:56:50.008 DEBUG:teuthology.orchestra.run.smithi176:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T05:56:50.036 INFO:tasks.cephadm:Stopping all daemons... 2024-02-10T05:56:50.037 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-02-10T05:56:50.037 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@mon.a 2024-02-10T05:56:50.193 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:50 smithi064 systemd[1]: Stopping Ceph mon.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:56:50.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:50 smithi176 ceph-mon[30294]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:56:50.488 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:50 smithi176 ceph-mon[30294]: from='client.24647 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:50.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:50 smithi064 ceph-mon[25185]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:56:50.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:50 smithi064 ceph-mon[25185]: from='client.24647 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:50.626 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:50 smithi064 ceph-mon[38958]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-10T05:56:50.626 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:50 smithi064 ceph-mon[38958]: from='client.24647 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:56:51.258 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:50 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-a[25181]: 2024-02-10T05:56:50.893+0000 7f89bbd92700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:56:51.258 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:50 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-a[25181]: 2024-02-10T05:56:50.893+0000 7f89bbd92700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-02-10T05:56:51.258 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:51 smithi064 podman[84793]: 2024-02-10 05:56:51.05668852 +0000 UTC m=+0.281628657 container died 22a0019eee341f33a7f0065792d0be56b53f5fae6d028ec29d471d635912da54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-a, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS) 2024-02-10T05:56:51.258 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:50 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: [10/Feb/2024:05:56:50] ENGINE Bus STOPPING 2024-02-10T05:56:51.258 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:50 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: [10/Feb/2024:05:56:50] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T05:56:51.259 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:50 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: [10/Feb/2024:05:56:50] ENGINE Bus STOPPED 2024-02-10T05:56:51.259 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:50 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: [10/Feb/2024:05:56:50] ENGINE Bus STARTING 2024-02-10T05:56:51.259 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:51 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: [10/Feb/2024:05:56:51] ENGINE Serving on http://:::9283 2024-02-10T05:56:51.259 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:51 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: [10/Feb/2024:05:56:51] ENGINE Bus STARTED 2024-02-10T05:56:51.259 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:51 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: ::ffff:172.21.15.176 - - [10/Feb/2024:05:56:51] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T05:56:51.519 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:51 smithi064 podman[84793]: 2024-02-10 05:56:51.519863852 +0000 UTC m=+0.744803997 container cleanup 22a0019eee341f33a7f0065792d0be56b53f5fae6d028ec29d471d635912da54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-02-10T05:56:51.876 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:51 smithi064 bash[84793]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-a 2024-02-10T05:56:52.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Feb 10 05:56:52 smithi064 podman[84904]: 2024-02-10 05:56:52.023312449 +0000 UTC m=+0.962978406 container remove 22a0019eee341f33a7f0065792d0be56b53f5fae6d028ec29d471d635912da54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS) 2024-02-10T05:56:52.848 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@mon.a.service' 2024-02-10T05:56:52.908 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:56:52.908 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-02-10T05:56:52.908 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-02-10T05:56:52.909 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@mon.c 2024-02-10T05:56:53.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:52 smithi064 systemd[1]: Stopping Ceph mon.c for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:56:53.867 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:53 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-c[38952]: 2024-02-10T05:56:53.574+0000 7f5d752e6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:56:53.868 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:53 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-c[38952]: 2024-02-10T05:56:53.574+0000 7f5d752e6700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-02-10T05:56:53.868 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:53 smithi064 podman[85167]: 2024-02-10 05:56:53.596171277 +0000 UTC m=+0.140682486 container died de9d442387609cab7fa63eef16c797782ca95d8be20845a3dfc6c14573efb88f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-c, org.label-schema.license=GPLv2, RELEASE=main-290212a, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-02-10T05:56:53.868 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:53 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: [10/Feb/2024:05:56:53] ENGINE Bus STOPPING 2024-02-10T05:56:53.868 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:53 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: [10/Feb/2024:05:56:53] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T05:56:53.868 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:53 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: [10/Feb/2024:05:56:53] ENGINE Bus STOPPED 2024-02-10T05:56:53.868 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:53 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: [10/Feb/2024:05:56:53] ENGINE Bus STARTING 2024-02-10T05:56:53.868 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:53 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: [10/Feb/2024:05:56:53] ENGINE Serving on http://:::9283 2024-02-10T05:56:53.868 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:53 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y[25421]: [10/Feb/2024:05:56:53] ENGINE Bus STARTED 2024-02-10T05:56:54.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:53 smithi064 podman[85167]: 2024-02-10 05:56:53.966751327 +0000 UTC m=+0.511262532 container cleanup de9d442387609cab7fa63eef16c797782ca95d8be20845a3dfc6c14573efb88f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-c, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-10T05:56:54.126 INFO:journalctl@ceph.mon.c.smithi064.stdout:Feb 10 05:56:53 smithi064 bash[85167]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-c 2024-02-10T05:56:54.399 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@mon.c.service' 2024-02-10T05:56:54.437 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:56:54.438 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-02-10T05:56:54.438 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-02-10T05:56:54.438 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@mon.b 2024-02-10T05:56:54.738 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:54 smithi176 systemd[1]: Stopping Ceph mon.b for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:56:55.119 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:54 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-b[30277]: 2024-02-10T05:56:54.761+0000 7f6dee892700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:56:55.119 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:54 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-b[30277]: 2024-02-10T05:56:54.761+0000 7f6dee892700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-02-10T05:56:55.119 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:54 smithi176 podman[70698]: 2024-02-10 05:56:54.853792606 +0000 UTC m=+0.207234308 container died 87e8c2896dc7f15f6e0639bd26a5eec0f71cdee19514dd3825cd404bb8554a9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a) 2024-02-10T05:56:55.395 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:55 smithi176 podman[70698]: 2024-02-10 05:56:55.201905445 +0000 UTC m=+0.555347153 container cleanup 87e8c2896dc7f15f6e0639bd26a5eec0f71cdee19514dd3825cd404bb8554a9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS) 2024-02-10T05:56:55.395 INFO:journalctl@ceph.mon.b.smithi176.stdout:Feb 10 05:56:55 smithi176 bash[70698]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mon-b 2024-02-10T05:56:55.524 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@mon.b.service' 2024-02-10T05:56:55.572 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:56:55.573 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-02-10T05:56:55.573 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-02-10T05:56:55.573 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@mgr.y 2024-02-10T05:56:55.876 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:55 smithi064 systemd[1]: Stopping Ceph mgr.y for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:56:56.188 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:55 smithi064 podman[85299]: 2024-02-10 05:56:55.91895255 +0000 UTC m=+0.149872813 container died 0209d0317adfbf50ed7401e8858d1298291832ddb496647360c139528f9e3bf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131) 2024-02-10T05:56:56.555 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:56 smithi064 podman[85299]: 2024-02-10 05:56:56.263022563 +0000 UTC m=+0.493942824 container cleanup 0209d0317adfbf50ed7401e8858d1298291832ddb496647360c139528f9e3bf3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.build-date=20240131) 2024-02-10T05:56:56.555 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 10 05:56:56 smithi064 bash[85299]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-y 2024-02-10T05:56:56.690 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@mgr.y.service' 2024-02-10T05:56:56.727 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:56:56.727 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-02-10T05:56:56.727 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-02-10T05:56:56.728 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@mgr.x 2024-02-10T05:56:57.071 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:56:56 smithi176 systemd[1]: Stopping Ceph mgr.x for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:56:57.324 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:56:57 smithi176 podman[70820]: 2024-02-10 05:56:57.071539675 +0000 UTC m=+0.152953128 container died 0404dd01867ca4247fa872a9037d4ca5b48b2969383d506a9893f8771588909a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-02-10T05:56:57.738 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:56:57 smithi176 podman[70820]: 2024-02-10 05:56:57.431905638 +0000 UTC m=+0.513319089 container cleanup 0404dd01867ca4247fa872a9037d4ca5b48b2969383d506a9893f8771588909a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=main-290212a, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-02-10T05:56:57.738 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Feb 10 05:56:57 smithi176 bash[70820]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-mgr-x 2024-02-10T05:56:57.856 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@mgr.x.service' 2024-02-10T05:56:57.893 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:56:57.894 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-02-10T05:56:57.894 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-10T05:56:57.894 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.0 2024-02-10T05:56:58.209 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:56:57 smithi064 systemd[1]: Stopping Ceph osd.0 for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:56:58.626 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:56:58 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-0[42822]: 2024-02-10T05:56:58.208+0000 7f41f34d4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:56:58.626 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:56:58 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-0[42822]: 2024-02-10T05:56:58.208+0000 7f41f34d4700 -1 osd.0 71 *** Got signal Terminated *** 2024-02-10T05:56:58.626 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:56:58 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-0[42822]: 2024-02-10T05:56:58.208+0000 7f41f34d4700 -1 osd.0 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:57:03.869 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:57:03 smithi064 podman[85420]: 2024-02-10 05:57:03.570612644 +0000 UTC m=+5.485594830 container died c5e21e6393d6b3039de3812373c80c28633729f4075c4c253f1d4001345a82d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-290212a, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-02-10T05:57:04.126 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:57:04 smithi064 podman[85420]: 2024-02-10 05:57:04.017639565 +0000 UTC m=+5.932621800 container cleanup c5e21e6393d6b3039de3812373c80c28633729f4075c4c253f1d4001345a82d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-02-10T05:57:04.126 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:57:04 smithi064 bash[85420]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-0 2024-02-10T05:57:04.626 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:57:04 smithi064 podman[85437]: 2024-02-10 05:57:04.302650748 +0000 UTC m=+0.725657178 container remove c5e21e6393d6b3039de3812373c80c28633729f4075c4c253f1d4001345a82d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T05:57:05.042 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:57:04 smithi064 podman[85527]: 2024-02-10 05:57:04.69354874 +0000 UTC m=+0.018593579 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:57:05.042 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:57:04 smithi064 podman[85527]: 2024-02-10 05:57:04.79524261 +0000 UTC m=+0.120287432 container create c4a89b2c3f00941758dceb26b10a78224bb6f906f68f75f15fe600021156cfab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-0-deactivate, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-02-10T05:57:05.042 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:57:05 smithi064 podman[85527]: 2024-02-10 05:57:05.03949648 +0000 UTC m=+0.364541314 container init c4a89b2c3f00941758dceb26b10a78224bb6f906f68f75f15fe600021156cfab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0) 2024-02-10T05:57:05.042 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:57:05 smithi064 podman[85527]: 2024-02-10 05:57:05.042646353 +0000 UTC m=+0.367691179 container start c4a89b2c3f00941758dceb26b10a78224bb6f906f68f75f15fe600021156cfab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-290212a) 2024-02-10T05:57:05.351 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:57:05 smithi064 podman[85527]: 2024-02-10 05:57:05.139464049 +0000 UTC m=+0.464508907 container attach c4a89b2c3f00941758dceb26b10a78224bb6f906f68f75f15fe600021156cfab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131) 2024-02-10T05:57:05.626 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:57:05 smithi064 podman[85527]: 2024-02-10 05:57:05.352202696 +0000 UTC m=+0.677247519 container died c4a89b2c3f00941758dceb26b10a78224bb6f906f68f75f15fe600021156cfab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:57:05.970 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.0.service' 2024-02-10T05:57:05.979 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:57:05 smithi064 podman[85554]: 2024-02-10 05:57:05.95149699 +0000 UTC m=+0.593597819 container remove c4a89b2c3f00941758dceb26b10a78224bb6f906f68f75f15fe600021156cfab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-02-10T05:57:05.979 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:57:05 smithi064 systemd[1]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.0.service: Deactivated successfully. 2024-02-10T05:57:05.979 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:57:05 smithi064 systemd[1]: Stopped Ceph osd.0 for 00e86dbe-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:57:05.979 INFO:journalctl@ceph.osd.0.smithi064.stdout:Feb 10 05:57:05 smithi064 systemd[1]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.0.service: Consumed 4.158s CPU time. 2024-02-10T05:57:06.016 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:57:06.016 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-10T05:57:06.016 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-10T05:57:06.016 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.1 2024-02-10T05:57:06.239 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 05:57:06 smithi064 systemd[1]: Stopping Ceph osd.1 for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:57:06.626 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 05:57:06 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-1[50712]: 2024-02-10T05:57:06.345+0000 7f3ec9fd5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:57:06.626 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 05:57:06 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-1[50712]: 2024-02-10T05:57:06.345+0000 7f3ec9fd5700 -1 osd.1 71 *** Got signal Terminated *** 2024-02-10T05:57:06.626 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 05:57:06 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-1[50712]: 2024-02-10T05:57:06.345+0000 7f3ec9fd5700 -1 osd.1 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:57:12.064 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 05:57:11 smithi064 podman[85644]: 2024-02-10 05:57:11.777211455 +0000 UTC m=+5.533037948 container died 8b4cf1877c697c7dabe0a6ab3f8ba12baa840b44c2444068710a4712cf255992 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-10T05:57:12.376 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 05:57:12 smithi064 podman[85644]: 2024-02-10 05:57:12.237722248 +0000 UTC m=+5.993548740 container cleanup 8b4cf1877c697c7dabe0a6ab3f8ba12baa840b44c2444068710a4712cf255992 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-1, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-290212a, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.license=GPLv2) 2024-02-10T05:57:12.376 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 05:57:12 smithi064 bash[85644]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-1 2024-02-10T05:57:12.876 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 05:57:12 smithi064 podman[85661]: 2024-02-10 05:57:12.564381813 +0000 UTC m=+0.780780405 container remove 8b4cf1877c697c7dabe0a6ab3f8ba12baa840b44c2444068710a4712cf255992 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-1, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, maintainer=Guillaume Abrioux ) 2024-02-10T05:57:13.205 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 05:57:12 smithi064 podman[85751]: 2024-02-10 05:57:12.89844321 +0000 UTC m=+0.106747842 container create 3eb63933c4931c71ec58518c1b5d08e35c42be7c71a8f3d44f921d0d32695cb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-1-deactivate, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.build-date=20240131, RELEASE=main-290212a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-02-10T05:57:13.205 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 05:57:12 smithi064 podman[85751]: 2024-02-10 05:57:12.809218833 +0000 UTC m=+0.017523419 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:57:13.205 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 05:57:13 smithi064 podman[85751]: 2024-02-10 05:57:13.109392893 +0000 UTC m=+0.317697472 container init 3eb63933c4931c71ec58518c1b5d08e35c42be7c71a8f3d44f921d0d32695cb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=) 2024-02-10T05:57:13.206 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 05:57:13 smithi064 podman[85751]: 2024-02-10 05:57:13.112519187 +0000 UTC m=+0.320823770 container start 3eb63933c4931c71ec58518c1b5d08e35c42be7c71a8f3d44f921d0d32695cb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS) 2024-02-10T05:57:13.626 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 05:57:13 smithi064 podman[85751]: 2024-02-10 05:57:13.204935723 +0000 UTC m=+0.413240314 container attach 3eb63933c4931c71ec58518c1b5d08e35c42be7c71a8f3d44f921d0d32695cb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-1-deactivate, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-02-10T05:57:13.626 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 10 05:57:13 smithi064 podman[85751]: 2024-02-10 05:57:13.411951571 +0000 UTC m=+0.620256145 container died 3eb63933c4931c71ec58518c1b5d08e35c42be7c71a8f3d44f921d0d32695cb7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T05:57:14.080 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.1.service' 2024-02-10T05:57:14.124 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:57:14.124 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-10T05:57:14.124 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-10T05:57:14.124 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.2 2024-02-10T05:57:14.376 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 05:57:14 smithi064 systemd[1]: Stopping Ceph osd.2 for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:57:14.879 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 05:57:14 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-2[58170]: 2024-02-10T05:57:14.468+0000 7fc48dc24700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:57:14.879 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 05:57:14 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-2[58170]: 2024-02-10T05:57:14.468+0000 7fc48dc24700 -1 osd.2 71 *** Got signal Terminated *** 2024-02-10T05:57:14.879 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 05:57:14 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-2[58170]: 2024-02-10T05:57:14.468+0000 7fc48dc24700 -1 osd.2 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:57:20.229 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 05:57:19 smithi064 podman[85856]: 2024-02-10 05:57:19.95959162 +0000 UTC m=+5.609796734 container died 3652b514b479aba91a6f18da6c73a35a54cbffc657dc1e590e35d0a6a143b864 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-2, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-10T05:57:20.626 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 05:57:20 smithi064 podman[85856]: 2024-02-10 05:57:20.42861152 +0000 UTC m=+6.078816629 container cleanup 3652b514b479aba91a6f18da6c73a35a54cbffc657dc1e590e35d0a6a143b864 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-2, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, ceph=True, org.label-schema.vendor=CentOS) 2024-02-10T05:57:20.626 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 05:57:20 smithi064 bash[85856]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-2 2024-02-10T05:57:21.046 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 05:57:20 smithi064 podman[85874]: 2024-02-10 05:57:20.730155554 +0000 UTC m=+0.761661043 container remove 3652b514b479aba91a6f18da6c73a35a54cbffc657dc1e590e35d0a6a143b864 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=) 2024-02-10T05:57:21.325 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 05:57:21 smithi064 podman[85965]: 2024-02-10 05:57:20.949485091 +0000 UTC m=+0.017676062 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:57:21.325 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 05:57:21 smithi064 podman[85965]: 2024-02-10 05:57:21.047429058 +0000 UTC m=+0.115620014 container create 5d5d6793277329a239c378240de5d5739a9dfc9c8ec8f510a17ce55f2a4d82f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=main-290212a, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:57:21.325 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 05:57:21 smithi064 podman[85965]: 2024-02-10 05:57:21.233251591 +0000 UTC m=+0.301442561 container init 5d5d6793277329a239c378240de5d5739a9dfc9c8ec8f510a17ce55f2a4d82f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.schema-version=1.0) 2024-02-10T05:57:21.325 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 05:57:21 smithi064 podman[85965]: 2024-02-10 05:57:21.236476603 +0000 UTC m=+0.304667572 container start 5d5d6793277329a239c378240de5d5739a9dfc9c8ec8f510a17ce55f2a4d82f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True) 2024-02-10T05:57:21.325 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 05:57:21 smithi064 podman[85965]: 2024-02-10 05:57:21.325037063 +0000 UTC m=+0.393228049 container attach 5d5d6793277329a239c378240de5d5739a9dfc9c8ec8f510a17ce55f2a4d82f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True) 2024-02-10T05:57:21.626 INFO:journalctl@ceph.osd.2.smithi064.stdout:Feb 10 05:57:21 smithi064 podman[85965]: 2024-02-10 05:57:21.531254809 +0000 UTC m=+0.599445768 container died 5d5d6793277329a239c378240de5d5739a9dfc9c8ec8f510a17ce55f2a4d82f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, RELEASE=main-290212a, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2) 2024-02-10T05:57:22.107 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.2.service' 2024-02-10T05:57:22.148 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:57:22.148 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-10T05:57:22.149 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-10T05:57:22.149 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.3 2024-02-10T05:57:22.353 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 05:57:22 smithi064 systemd[1]: Stopping Ceph osd.3 for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:57:22.626 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 05:57:22 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-3[66346]: 2024-02-10T05:57:22.506+0000 7f0f5583b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:57:22.626 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 05:57:22 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-3[66346]: 2024-02-10T05:57:22.506+0000 7f0f5583b700 -1 osd.3 71 *** Got signal Terminated *** 2024-02-10T05:57:22.626 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 05:57:22 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-3[66346]: 2024-02-10T05:57:22.506+0000 7f0f5583b700 -1 osd.3 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:57:24.126 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 05:57:23 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-3[66346]: 2024-02-10T05:57:23.794+0000 7f0f4de27700 -1 osd.3 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:59.540763+0000 front 2024-02-10T05:56:59.540801+0000 (oldest deadline 2024-02-10T05:57:23.640732+0000) 2024-02-10T05:57:25.126 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 05:57:24 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-3[66346]: 2024-02-10T05:57:24.767+0000 7f0f4de27700 -1 osd.3 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:59.540763+0000 front 2024-02-10T05:56:59.540801+0000 (oldest deadline 2024-02-10T05:57:23.640732+0000) 2024-02-10T05:57:25.238 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:24 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:24.937+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:26.126 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 05:57:25 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-3[66346]: 2024-02-10T05:57:25.771+0000 7f0f4de27700 -1 osd.3 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:59.540763+0000 front 2024-02-10T05:56:59.540801+0000 (oldest deadline 2024-02-10T05:57:23.640732+0000) 2024-02-10T05:57:26.238 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:25 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:25.972+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:27.126 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 05:57:26 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-3[66346]: 2024-02-10T05:57:26.731+0000 7f0f4de27700 -1 osd.3 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:59.540763+0000 front 2024-02-10T05:56:59.540801+0000 (oldest deadline 2024-02-10T05:57:23.640732+0000) 2024-02-10T05:57:27.238 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:26 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:26.940+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:27.738 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:27 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:27.397+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:27.738 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Feb 10 05:57:27 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-prometheus-a[67842]: ts=2024-02-10T05:57:27.309Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.64:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.64:8765: connect: connection refused" 2024-02-10T05:57:27.738 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Feb 10 05:57:27 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-prometheus-a[67842]: ts=2024-02-10T05:57:27.309Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.64:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.64:8765: connect: connection refused" 2024-02-10T05:57:28.238 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:27 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:27.917+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:28.273 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 05:57:28 smithi064 podman[86072]: 2024-02-10 05:57:28.006433397 +0000 UTC m=+5.620003819 container died 0b10f0df19847c5e32391a3740b9d0388bc369ce4f8faf9ddcac9a34c59086cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-3, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-02-10T05:57:28.626 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 05:57:28 smithi064 podman[86072]: 2024-02-10 05:57:28.422894091 +0000 UTC m=+6.036464554 container cleanup 0b10f0df19847c5e32391a3740b9d0388bc369ce4f8faf9ddcac9a34c59086cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-3, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-290212a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:57:28.626 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 05:57:28 smithi064 bash[86072]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-3 2024-02-10T05:57:28.738 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:28 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:28.240+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:58.632202+0000 front 2024-02-10T05:56:58.632055+0000 (oldest deadline 2024-02-10T05:57:27.432453+0000) 2024-02-10T05:57:28.738 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:28 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:28.410+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:29.067 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 05:57:28 smithi064 podman[86100]: 2024-02-10 05:57:28.657732011 +0000 UTC m=+0.652660494 container remove 0b10f0df19847c5e32391a3740b9d0388bc369ce4f8faf9ddcac9a34c59086cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-3, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=main-290212a) 2024-02-10T05:57:29.195 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:28 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:28.907+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:29.376 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 05:57:29 smithi064 podman[86191]: 2024-02-10 05:57:28.970092688 +0000 UTC m=+0.017199671 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:57:29.376 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 05:57:29 smithi064 podman[86191]: 2024-02-10 05:57:29.075151075 +0000 UTC m=+0.122258096 container create 0e7eb5796d76b7fd74271effdd9d7cb6217036cec048b3ec6d9703a1dddc754f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-290212a, ceph=True, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-02-10T05:57:29.376 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 05:57:29 smithi064 podman[86191]: 2024-02-10 05:57:29.295424884 +0000 UTC m=+0.342531871 container init 0e7eb5796d76b7fd74271effdd9d7cb6217036cec048b3ec6d9703a1dddc754f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-02-10T05:57:29.376 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 05:57:29 smithi064 podman[86191]: 2024-02-10 05:57:29.298728912 +0000 UTC m=+0.345835888 container start 0e7eb5796d76b7fd74271effdd9d7cb6217036cec048b3ec6d9703a1dddc754f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, maintainer=Guillaume Abrioux ) 2024-02-10T05:57:29.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:29 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:29.391+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:29.488 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:29 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:29.195+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:58.632202+0000 front 2024-02-10T05:56:58.632055+0000 (oldest deadline 2024-02-10T05:57:27.432453+0000) 2024-02-10T05:57:29.876 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 05:57:29 smithi064 podman[86191]: 2024-02-10 05:57:29.377865411 +0000 UTC m=+0.424972402 container attach 0e7eb5796d76b7fd74271effdd9d7cb6217036cec048b3ec6d9703a1dddc754f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.license=GPLv2) 2024-02-10T05:57:29.879 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 05:57:29 smithi064 podman[86191]: 2024-02-10 05:57:29.610529094 +0000 UTC m=+0.657636070 container died 0e7eb5796d76b7fd74271effdd9d7cb6217036cec048b3ec6d9703a1dddc754f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, RELEASE=main-290212a, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131) 2024-02-10T05:57:30.157 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:29 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:29.905+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:30.162 INFO:journalctl@ceph.osd.3.smithi064.stdout:Feb 10 05:57:30 smithi064 podman[86218]: 2024-02-10 05:57:30.139385951 +0000 UTC m=+0.524344267 container remove 0e7eb5796d76b7fd74271effdd9d7cb6217036cec048b3ec6d9703a1dddc754f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, ceph=True, RELEASE=main-290212a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-02-10T05:57:30.167 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.3.service' 2024-02-10T05:57:30.207 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:57:30.208 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-10T05:57:30.208 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-10T05:57:30.208 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.4 2024-02-10T05:57:30.488 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:57:30 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-4[35167]: 2024-02-10T05:57:30.157+0000 7f5fe8b94700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:58.114124+0000 front 2024-02-10T05:56:58.114060+0000 (oldest deadline 2024-02-10T05:57:29.314567+0000) 2024-02-10T05:57:30.488 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:57:30 smithi176 systemd[1]: Stopping Ceph osd.4 for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:57:30.489 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:30 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:30.230+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:58.632202+0000 front 2024-02-10T05:56:58.632055+0000 (oldest deadline 2024-02-10T05:57:27.432453+0000) 2024-02-10T05:57:30.489 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:30 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:30.379+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:30.934 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:57:30 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-4[35167]: 2024-02-10T05:57:30.505+0000 7f5fec5a0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:57:30.934 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:57:30 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-4[35167]: 2024-02-10T05:57:30.505+0000 7f5fec5a0700 -1 osd.4 71 *** Got signal Terminated *** 2024-02-10T05:57:30.934 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:57:30 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-4[35167]: 2024-02-10T05:57:30.505+0000 7f5fec5a0700 -1 osd.4 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:57:31.238 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:57:31 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-4[35167]: 2024-02-10T05:57:31.163+0000 7f5fe8b94700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:58.114124+0000 front 2024-02-10T05:56:58.114060+0000 (oldest deadline 2024-02-10T05:57:29.314567+0000) 2024-02-10T05:57:31.238 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:31 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:31.183+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:58.632202+0000 front 2024-02-10T05:56:58.632055+0000 (oldest deadline 2024-02-10T05:57:27.432453+0000) 2024-02-10T05:57:31.239 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:30 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:30.933+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:31.738 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:31 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:31.364+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:32.204 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:57:32 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-4[35167]: 2024-02-10T05:57:32.113+0000 7f5fe8b94700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:58.114124+0000 front 2024-02-10T05:56:58.114060+0000 (oldest deadline 2024-02-10T05:57:29.314567+0000) 2024-02-10T05:57:32.205 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:31 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:31.914+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:32.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:32 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:32.329+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:32.488 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:32 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:32.204+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:58.632202+0000 front 2024-02-10T05:56:58.632055+0000 (oldest deadline 2024-02-10T05:57:27.432453+0000) 2024-02-10T05:57:33.238 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:57:33 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-4[35167]: 2024-02-10T05:57:33.119+0000 7f5fe8b94700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:58.114124+0000 front 2024-02-10T05:56:58.114060+0000 (oldest deadline 2024-02-10T05:57:29.314567+0000) 2024-02-10T05:57:33.238 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:32 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:32.921+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:33.738 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:33 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:33.245+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:58.632202+0000 front 2024-02-10T05:56:58.632055+0000 (oldest deadline 2024-02-10T05:57:27.432453+0000) 2024-02-10T05:57:33.738 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:33 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:33.285+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:33.739 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:33 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:33.285+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:09.478827+0000 front 2024-02-10T05:57:09.478815+0000 (oldest deadline 2024-02-10T05:57:32.378782+0000) 2024-02-10T05:57:34.238 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:33 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:33.966+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:34.238 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:57:34 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-4[35167]: 2024-02-10T05:57:34.108+0000 7f5fe8b94700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:58.114124+0000 front 2024-02-10T05:56:58.114060+0000 (oldest deadline 2024-02-10T05:57:29.314567+0000) 2024-02-10T05:57:34.239 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:57:34 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-4[35167]: 2024-02-10T05:57:34.108+0000 7f5fe8b94700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:09.816050+0000 front 2024-02-10T05:57:09.815934+0000 (oldest deadline 2024-02-10T05:57:33.315797+0000) 2024-02-10T05:57:34.738 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:34 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:34.239+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:34.738 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:34 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:34.239+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:09.478827+0000 front 2024-02-10T05:57:09.478815+0000 (oldest deadline 2024-02-10T05:57:32.378782+0000) 2024-02-10T05:57:34.739 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:34 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:34.290+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:58.632202+0000 front 2024-02-10T05:56:58.632055+0000 (oldest deadline 2024-02-10T05:57:27.432453+0000) 2024-02-10T05:57:35.238 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:34 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:34.969+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:35.238 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:57:35 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-4[35167]: 2024-02-10T05:57:35.069+0000 7f5fe8b94700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:58.114124+0000 front 2024-02-10T05:56:58.114060+0000 (oldest deadline 2024-02-10T05:57:29.314567+0000) 2024-02-10T05:57:35.239 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:57:35 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-4[35167]: 2024-02-10T05:57:35.069+0000 7f5fe8b94700 -1 osd.4 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:09.816050+0000 front 2024-02-10T05:57:09.815934+0000 (oldest deadline 2024-02-10T05:57:33.315797+0000) 2024-02-10T05:57:35.239 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:35 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:35.202+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:35.239 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:35 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:35.202+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:09.478827+0000 front 2024-02-10T05:57:09.478815+0000 (oldest deadline 2024-02-10T05:57:32.378782+0000) 2024-02-10T05:57:35.738 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:35 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:35.303+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:58.632202+0000 front 2024-02-10T05:56:58.632055+0000 (oldest deadline 2024-02-10T05:57:27.432453+0000) 2024-02-10T05:57:35.739 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:35 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:35.303+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.333598+0000 front 2024-02-10T05:57:10.333509+0000 (oldest deadline 2024-02-10T05:57:35.033369+0000) 2024-02-10T05:57:36.181 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:57:35 smithi176 podman[70957]: 2024-02-10 05:57:35.939671161 +0000 UTC m=+5.537930988 container died 9b73b664d9b516012de383fa899412366201411af67cbafcacb9f1f251706f88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-4, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-02-10T05:57:36.181 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:35 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:35.934+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:36.450 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:57:36 smithi176 podman[70957]: 2024-02-10 05:57:36.314291933 +0000 UTC m=+5.912551759 container cleanup 9b73b664d9b516012de383fa899412366201411af67cbafcacb9f1f251706f88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-4, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-02-10T05:57:36.450 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:57:36 smithi176 bash[70957]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-4 2024-02-10T05:57:36.451 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:36 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:36.181+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:36.451 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:36 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:36.181+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:09.478827+0000 front 2024-02-10T05:57:09.478815+0000 (oldest deadline 2024-02-10T05:57:32.378782+0000) 2024-02-10T05:57:36.451 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:36 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:36.336+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:58.632202+0000 front 2024-02-10T05:56:58.632055+0000 (oldest deadline 2024-02-10T05:57:27.432453+0000) 2024-02-10T05:57:36.451 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:36 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:36.336+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.333598+0000 front 2024-02-10T05:57:10.333509+0000 (oldest deadline 2024-02-10T05:57:35.033369+0000) 2024-02-10T05:57:36.738 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:57:36 smithi176 podman[70974]: 2024-02-10 05:57:36.573946218 +0000 UTC m=+0.632283630 container remove 9b73b664d9b516012de383fa899412366201411af67cbafcacb9f1f251706f88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-4, GIT_CLEAN=True, RELEASE=main-290212a, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-02-10T05:57:37.196 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:36 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:36.946+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:37.196 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:36 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:36.946+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.372516+0000 front 2024-02-10T05:57:10.372572+0000 (oldest deadline 2024-02-10T05:57:36.272344+0000) 2024-02-10T05:57:37.196 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:57:36 smithi176 podman[71068]: 2024-02-10 05:57:36.916386834 +0000 UTC m=+0.107474924 container create ae93bff08bf51d98ed54608d47799d564d5f855f61c15144321122ccdb7a5412 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-4-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-290212a) 2024-02-10T05:57:37.197 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:57:36 smithi176 podman[71068]: 2024-02-10 05:57:36.826881725 +0000 UTC m=+0.017969820 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:57:37.197 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:57:37 smithi176 podman[71068]: 2024-02-10 05:57:37.127028381 +0000 UTC m=+0.318116472 container init ae93bff08bf51d98ed54608d47799d564d5f855f61c15144321122ccdb7a5412 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, ceph=True) 2024-02-10T05:57:37.197 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:57:37 smithi176 podman[71068]: 2024-02-10 05:57:37.130228608 +0000 UTC m=+0.321316699 container start ae93bff08bf51d98ed54608d47799d564d5f855f61c15144321122ccdb7a5412 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_CLEAN=True) 2024-02-10T05:57:37.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:37 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:37.195+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:37.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:37 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:37.195+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:09.478827+0000 front 2024-02-10T05:57:09.478815+0000 (oldest deadline 2024-02-10T05:57:32.378782+0000) 2024-02-10T05:57:37.489 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:57:37 smithi176 podman[71068]: 2024-02-10 05:57:37.236044086 +0000 UTC m=+0.427132189 container attach ae93bff08bf51d98ed54608d47799d564d5f855f61c15144321122ccdb7a5412 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=) 2024-02-10T05:57:37.489 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:57:37 smithi176 podman[71068]: 2024-02-10 05:57:37.41879677 +0000 UTC m=+0.609884861 container died ae93bff08bf51d98ed54608d47799d564d5f855f61c15144321122ccdb7a5412 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T05:57:37.489 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:37 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:37.300+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:58.632202+0000 front 2024-02-10T05:56:58.632055+0000 (oldest deadline 2024-02-10T05:57:27.432453+0000) 2024-02-10T05:57:37.489 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:37 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:37.300+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.333598+0000 front 2024-02-10T05:57:10.333509+0000 (oldest deadline 2024-02-10T05:57:35.033369+0000) 2024-02-10T05:57:37.932 INFO:journalctl@ceph.osd.4.smithi176.stdout:Feb 10 05:57:37 smithi176 podman[71094]: 2024-02-10 05:57:37.909596808 +0000 UTC m=+0.484946465 container remove ae93bff08bf51d98ed54608d47799d564d5f855f61c15144321122ccdb7a5412 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-02-10T05:57:37.933 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:37 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:37.919+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:37.933 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:37 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:37.919+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.372516+0000 front 2024-02-10T05:57:10.372572+0000 (oldest deadline 2024-02-10T05:57:36.272344+0000) 2024-02-10T05:57:37.937 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.4.service' 2024-02-10T05:57:37.979 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:57:37.979 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-10T05:57:37.979 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-10T05:57:37.979 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.5 2024-02-10T05:57:38.210 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:38 smithi176 systemd[1]: Stopping Ceph osd.5 for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:57:38.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:38 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:38.209+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:38.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:38 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:38.209+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:09.478827+0000 front 2024-02-10T05:57:09.478815+0000 (oldest deadline 2024-02-10T05:57:32.378782+0000) 2024-02-10T05:57:38.488 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:38 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:38.268+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:58.632202+0000 front 2024-02-10T05:56:58.632055+0000 (oldest deadline 2024-02-10T05:57:27.432453+0000) 2024-02-10T05:57:38.489 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:38 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:38.268+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.333598+0000 front 2024-02-10T05:57:10.333509+0000 (oldest deadline 2024-02-10T05:57:35.033369+0000) 2024-02-10T05:57:38.489 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:38 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:38.349+0000 7f5e0990f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:57:38.489 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:38 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:38.349+0000 7f5e0990f700 -1 osd.5 71 *** Got signal Terminated *** 2024-02-10T05:57:38.489 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:38 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:38.349+0000 7f5e0990f700 -1 osd.5 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:57:39.188 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:39.172+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:39.188 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:39.172+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:09.478827+0000 front 2024-02-10T05:57:09.478815+0000 (oldest deadline 2024-02-10T05:57:32.378782+0000) 2024-02-10T05:57:39.188 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:38 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:38.928+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:39.188 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:38 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:38.928+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.372516+0000 front 2024-02-10T05:57:10.372572+0000 (oldest deadline 2024-02-10T05:57:36.272344+0000) 2024-02-10T05:57:39.488 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:39.310+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:58.632202+0000 front 2024-02-10T05:56:58.632055+0000 (oldest deadline 2024-02-10T05:57:27.432453+0000) 2024-02-10T05:57:39.488 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:39.310+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.333598+0000 front 2024-02-10T05:57:10.333509+0000 (oldest deadline 2024-02-10T05:57:35.033369+0000) 2024-02-10T05:57:40.238 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:40 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:40.162+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:40.238 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:40 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:40.162+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:09.478827+0000 front 2024-02-10T05:57:09.478815+0000 (oldest deadline 2024-02-10T05:57:32.378782+0000) 2024-02-10T05:57:40.239 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:39.913+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:40.239 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:39 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:39.913+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.372516+0000 front 2024-02-10T05:57:10.372572+0000 (oldest deadline 2024-02-10T05:57:36.272344+0000) 2024-02-10T05:57:40.738 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:40 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:40.293+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:58.632202+0000 front 2024-02-10T05:56:58.632055+0000 (oldest deadline 2024-02-10T05:57:27.432453+0000) 2024-02-10T05:57:40.738 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:40 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:40.293+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.333598+0000 front 2024-02-10T05:57:10.333509+0000 (oldest deadline 2024-02-10T05:57:35.033369+0000) 2024-02-10T05:57:41.177 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:40 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:40.925+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:41.177 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:40 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:40.925+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.372516+0000 front 2024-02-10T05:57:10.372572+0000 (oldest deadline 2024-02-10T05:57:36.272344+0000) 2024-02-10T05:57:41.488 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:41 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:41.267+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:58.632202+0000 front 2024-02-10T05:56:58.632055+0000 (oldest deadline 2024-02-10T05:57:27.432453+0000) 2024-02-10T05:57:41.488 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:41 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:41.267+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.333598+0000 front 2024-02-10T05:57:10.333509+0000 (oldest deadline 2024-02-10T05:57:35.033369+0000) 2024-02-10T05:57:41.488 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:41 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:41.267+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:15.034060+0000 front 2024-02-10T05:57:15.034092+0000 (oldest deadline 2024-02-10T05:57:40.333960+0000) 2024-02-10T05:57:41.489 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:41 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:41.176+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:41.489 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:41 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:41.176+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:09.478827+0000 front 2024-02-10T05:57:09.478815+0000 (oldest deadline 2024-02-10T05:57:32.378782+0000) 2024-02-10T05:57:41.489 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:41 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:41.176+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:18.080464+0000 front 2024-02-10T05:57:18.080558+0000 (oldest deadline 2024-02-10T05:57:40.380400+0000) 2024-02-10T05:57:42.238 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:41 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:41.902+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:42.238 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:41 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:41.902+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.372516+0000 front 2024-02-10T05:57:10.372572+0000 (oldest deadline 2024-02-10T05:57:36.272344+0000) 2024-02-10T05:57:42.238 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:42 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:42.149+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:42.239 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:42 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:42.149+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:09.478827+0000 front 2024-02-10T05:57:09.478815+0000 (oldest deadline 2024-02-10T05:57:32.378782+0000) 2024-02-10T05:57:42.239 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:42 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:42.149+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:18.080464+0000 front 2024-02-10T05:57:18.080558+0000 (oldest deadline 2024-02-10T05:57:40.380400+0000) 2024-02-10T05:57:42.738 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:42 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:42.307+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:58.632202+0000 front 2024-02-10T05:56:58.632055+0000 (oldest deadline 2024-02-10T05:57:27.432453+0000) 2024-02-10T05:57:42.738 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:42 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:42.307+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.333598+0000 front 2024-02-10T05:57:10.333509+0000 (oldest deadline 2024-02-10T05:57:35.033369+0000) 2024-02-10T05:57:42.738 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:42 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:42.307+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:15.034060+0000 front 2024-02-10T05:57:15.034092+0000 (oldest deadline 2024-02-10T05:57:40.333960+0000) 2024-02-10T05:57:43.197 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:42 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:42.893+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:43.197 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:42 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:42.893+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.372516+0000 front 2024-02-10T05:57:10.372572+0000 (oldest deadline 2024-02-10T05:57:36.272344+0000) 2024-02-10T05:57:43.197 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:42 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:42.893+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:16.272859+0000 front 2024-02-10T05:57:16.272813+0000 (oldest deadline 2024-02-10T05:57:42.172672+0000) 2024-02-10T05:57:43.197 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:42 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:42.893+0000 7f8a3d42a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24416.0:124 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-02-10T05:57:43.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:43 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:43.197+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:43.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:43 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:43.197+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:09.478827+0000 front 2024-02-10T05:57:09.478815+0000 (oldest deadline 2024-02-10T05:57:32.378782+0000) 2024-02-10T05:57:43.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:43 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:43.197+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:18.080464+0000 front 2024-02-10T05:57:18.080558+0000 (oldest deadline 2024-02-10T05:57:40.380400+0000) 2024-02-10T05:57:43.489 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:43 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:43.299+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:56:58.632202+0000 front 2024-02-10T05:56:58.632055+0000 (oldest deadline 2024-02-10T05:57:27.432453+0000) 2024-02-10T05:57:43.489 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:43 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:43.299+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.333598+0000 front 2024-02-10T05:57:10.333509+0000 (oldest deadline 2024-02-10T05:57:35.033369+0000) 2024-02-10T05:57:43.489 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:43 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5[42940]: 2024-02-10T05:57:43.299+0000 7f5e05f03700 -1 osd.5 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:15.034060+0000 front 2024-02-10T05:57:15.034092+0000 (oldest deadline 2024-02-10T05:57:40.333960+0000) 2024-02-10T05:57:44.201 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:43 smithi176 podman[71174]: 2024-02-10 05:57:43.809948439 +0000 UTC m=+5.588286122 container died b5886e6e8aecd674e12fe89f5b1a2f994fe7e5bfe2b573730e9780023a17802f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:57:44.201 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:43 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:43.933+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:44.201 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:43 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:43.933+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.372516+0000 front 2024-02-10T05:57:10.372572+0000 (oldest deadline 2024-02-10T05:57:36.272344+0000) 2024-02-10T05:57:44.202 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:43 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:43.933+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:16.272859+0000 front 2024-02-10T05:57:16.272813+0000 (oldest deadline 2024-02-10T05:57:42.172672+0000) 2024-02-10T05:57:44.202 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:43 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:43.933+0000 7f8a3d42a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24416.0:124 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-02-10T05:57:44.488 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:44 smithi176 podman[71174]: 2024-02-10 05:57:44.316683777 +0000 UTC m=+6.095021459 container cleanup b5886e6e8aecd674e12fe89f5b1a2f994fe7e5bfe2b573730e9780023a17802f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5, RELEASE=main-290212a, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-02-10T05:57:44.488 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:44 smithi176 bash[71174]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5 2024-02-10T05:57:44.489 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:44 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:44.201+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:44.489 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:44 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:44.201+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:09.478827+0000 front 2024-02-10T05:57:09.478815+0000 (oldest deadline 2024-02-10T05:57:32.378782+0000) 2024-02-10T05:57:44.489 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:44 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:44.201+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:18.080464+0000 front 2024-02-10T05:57:18.080558+0000 (oldest deadline 2024-02-10T05:57:40.380400+0000) 2024-02-10T05:57:44.844 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:44 smithi176 podman[71202]: 2024-02-10 05:57:44.567959247 +0000 UTC m=+0.753759822 container remove b5886e6e8aecd674e12fe89f5b1a2f994fe7e5bfe2b573730e9780023a17802f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a) 2024-02-10T05:57:45.095 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:44 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:44.919+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:45.096 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:44 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:44.919+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.372516+0000 front 2024-02-10T05:57:10.372572+0000 (oldest deadline 2024-02-10T05:57:36.272344+0000) 2024-02-10T05:57:45.096 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:44 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:44.920+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:16.272859+0000 front 2024-02-10T05:57:16.272813+0000 (oldest deadline 2024-02-10T05:57:42.172672+0000) 2024-02-10T05:57:45.096 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:44 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:44.920+0000 7f8a3d42a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24416.0:124 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-02-10T05:57:45.096 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:44 smithi176 podman[71292]: 2024-02-10 05:57:44.843705781 +0000 UTC m=+0.088928845 container create 6709c5b39fe95916f02c99579dbcb3ea067d989262bbb922e0485f8d2e6d9237 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-290212a) 2024-02-10T05:57:45.096 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:44 smithi176 podman[71292]: 2024-02-10 05:57:44.772477979 +0000 UTC m=+0.017701054 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:57:45.096 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:45 smithi176 podman[71292]: 2024-02-10 05:57:45.037673873 +0000 UTC m=+0.282896946 container init 6709c5b39fe95916f02c99579dbcb3ea067d989262bbb922e0485f8d2e6d9237 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-10T05:57:45.096 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:45 smithi176 podman[71292]: 2024-02-10 05:57:45.040947431 +0000 UTC m=+0.286170495 container start 6709c5b39fe95916f02c99579dbcb3ea067d989262bbb922e0485f8d2e6d9237 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, ceph=True) 2024-02-10T05:57:45.097 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:45 smithi176 podman[71292]: 2024-02-10 05:57:45.096187986 +0000 UTC m=+0.341411061 container attach 6709c5b39fe95916f02c99579dbcb3ea067d989262bbb922e0485f8d2e6d9237 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-02-10T05:57:45.488 INFO:journalctl@ceph.osd.5.smithi176.stdout:Feb 10 05:57:45 smithi176 podman[71292]: 2024-02-10 05:57:45.339643626 +0000 UTC m=+0.584866690 container died 6709c5b39fe95916f02c99579dbcb3ea067d989262bbb922e0485f8d2e6d9237 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True) 2024-02-10T05:57:45.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:45 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:45.153+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:45.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:45 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:45.153+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:09.478827+0000 front 2024-02-10T05:57:09.478815+0000 (oldest deadline 2024-02-10T05:57:32.378782+0000) 2024-02-10T05:57:45.489 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:45 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:45.153+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:18.080464+0000 front 2024-02-10T05:57:18.080558+0000 (oldest deadline 2024-02-10T05:57:40.380400+0000) 2024-02-10T05:57:45.922 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.5.service' 2024-02-10T05:57:45.964 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:57:45.965 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-10T05:57:45.965 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-10T05:57:45.965 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.6 2024-02-10T05:57:46.177 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:46 smithi176 systemd[1]: Stopping Ceph osd.6 for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:57:46.177 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:45 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:45.899+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:46.177 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:45 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:45.899+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.372516+0000 front 2024-02-10T05:57:10.372572+0000 (oldest deadline 2024-02-10T05:57:36.272344+0000) 2024-02-10T05:57:46.177 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:45 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:45.899+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:16.272859+0000 front 2024-02-10T05:57:16.272813+0000 (oldest deadline 2024-02-10T05:57:42.172672+0000) 2024-02-10T05:57:46.177 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:45 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:45.899+0000 7f8a3d42a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24416.0:124 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-02-10T05:57:46.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:46 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:46.176+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:46.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:46 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:46.176+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:09.478827+0000 front 2024-02-10T05:57:09.478815+0000 (oldest deadline 2024-02-10T05:57:32.378782+0000) 2024-02-10T05:57:46.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:46 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:46.176+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:18.080464+0000 front 2024-02-10T05:57:18.080558+0000 (oldest deadline 2024-02-10T05:57:40.380400+0000) 2024-02-10T05:57:46.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:46 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:46.285+0000 7f1c9a3c0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:57:46.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:46 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:46.285+0000 7f1c9a3c0700 -1 osd.6 71 *** Got signal Terminated *** 2024-02-10T05:57:46.489 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:46 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:46.285+0000 7f1c9a3c0700 -1 osd.6 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:57:47.165 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:46 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:46.864+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:47.165 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:46 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:46.864+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.372516+0000 front 2024-02-10T05:57:10.372572+0000 (oldest deadline 2024-02-10T05:57:36.272344+0000) 2024-02-10T05:57:47.165 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:46 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:46.864+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:16.272859+0000 front 2024-02-10T05:57:16.272813+0000 (oldest deadline 2024-02-10T05:57:42.172672+0000) 2024-02-10T05:57:47.165 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:46 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:46.864+0000 7f8a3d42a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24416.0:124 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-02-10T05:57:47.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:47 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:47.165+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:47.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:47 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:47.165+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:09.478827+0000 front 2024-02-10T05:57:09.478815+0000 (oldest deadline 2024-02-10T05:57:32.378782+0000) 2024-02-10T05:57:47.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:47 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:47.165+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:18.080464+0000 front 2024-02-10T05:57:18.080558+0000 (oldest deadline 2024-02-10T05:57:40.380400+0000) 2024-02-10T05:57:48.190 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:47 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:47.822+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:48.190 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:47 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:47.822+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.372516+0000 front 2024-02-10T05:57:10.372572+0000 (oldest deadline 2024-02-10T05:57:36.272344+0000) 2024-02-10T05:57:48.190 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:47 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:47.822+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:16.272859+0000 front 2024-02-10T05:57:16.272813+0000 (oldest deadline 2024-02-10T05:57:42.172672+0000) 2024-02-10T05:57:48.190 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:47 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:47.822+0000 7f8a3d42a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24416.0:124 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-02-10T05:57:48.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:48.190+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:48.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:48.190+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:09.478827+0000 front 2024-02-10T05:57:09.478815+0000 (oldest deadline 2024-02-10T05:57:32.378782+0000) 2024-02-10T05:57:48.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:48.190+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:18.080464+0000 front 2024-02-10T05:57:18.080558+0000 (oldest deadline 2024-02-10T05:57:40.380400+0000) 2024-02-10T05:57:48.988 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:48.837+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:48.988 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:48.837+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.372516+0000 front 2024-02-10T05:57:10.372572+0000 (oldest deadline 2024-02-10T05:57:36.272344+0000) 2024-02-10T05:57:48.988 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:48.837+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:16.272859+0000 front 2024-02-10T05:57:16.272813+0000 (oldest deadline 2024-02-10T05:57:42.172672+0000) 2024-02-10T05:57:48.989 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:48.837+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6830 osd.3 since back 2024-02-10T05:57:25.073701+0000 front 2024-02-10T05:57:25.073739+0000 (oldest deadline 2024-02-10T05:57:48.573659+0000) 2024-02-10T05:57:48.989 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:48 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:48.838+0000 7f8a3d42a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24416.0:124 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-02-10T05:57:49.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:49 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:49.154+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:49.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:49 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:49.154+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:09.478827+0000 front 2024-02-10T05:57:09.478815+0000 (oldest deadline 2024-02-10T05:57:32.378782+0000) 2024-02-10T05:57:49.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:49 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:49.154+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:18.080464+0000 front 2024-02-10T05:57:18.080558+0000 (oldest deadline 2024-02-10T05:57:40.380400+0000) 2024-02-10T05:57:50.174 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:49 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:49.856+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:50.174 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:49 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:49.856+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.372516+0000 front 2024-02-10T05:57:10.372572+0000 (oldest deadline 2024-02-10T05:57:36.272344+0000) 2024-02-10T05:57:50.174 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:49 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:49.856+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:16.272859+0000 front 2024-02-10T05:57:16.272813+0000 (oldest deadline 2024-02-10T05:57:42.172672+0000) 2024-02-10T05:57:50.174 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:49 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:49.856+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6830 osd.3 since back 2024-02-10T05:57:25.073701+0000 front 2024-02-10T05:57:25.073739+0000 (oldest deadline 2024-02-10T05:57:48.573659+0000) 2024-02-10T05:57:50.174 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:49 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:49.856+0000 7f8a3d42a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24416.0:124 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-02-10T05:57:50.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:50 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:50.174+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:50.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:50 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:50.174+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:09.478827+0000 front 2024-02-10T05:57:09.478815+0000 (oldest deadline 2024-02-10T05:57:32.378782+0000) 2024-02-10T05:57:50.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:50 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:50.174+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:18.080464+0000 front 2024-02-10T05:57:18.080558+0000 (oldest deadline 2024-02-10T05:57:40.380400+0000) 2024-02-10T05:57:50.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:50 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:50.174+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6830 osd.3 since back 2024-02-10T05:57:25.681180+0000 front 2024-02-10T05:57:25.681212+0000 (oldest deadline 2024-02-10T05:57:49.781079+0000) 2024-02-10T05:57:51.176 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:50 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:50.904+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:51.176 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:50 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:50.904+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.372516+0000 front 2024-02-10T05:57:10.372572+0000 (oldest deadline 2024-02-10T05:57:36.272344+0000) 2024-02-10T05:57:51.176 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:50 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:50.904+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:16.272859+0000 front 2024-02-10T05:57:16.272813+0000 (oldest deadline 2024-02-10T05:57:42.172672+0000) 2024-02-10T05:57:51.176 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:50 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:50.904+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6830 osd.3 since back 2024-02-10T05:57:25.073701+0000 front 2024-02-10T05:57:25.073739+0000 (oldest deadline 2024-02-10T05:57:48.573659+0000) 2024-02-10T05:57:51.176 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:50 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:50.905+0000 7f8a3d42a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24416.0:124 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-02-10T05:57:51.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:51 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:51.175+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.478088+0000 front 2024-02-10T05:57:02.478066+0000 (oldest deadline 2024-02-10T05:57:26.577792+0000) 2024-02-10T05:57:51.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:51 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:51.175+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:09.478827+0000 front 2024-02-10T05:57:09.478815+0000 (oldest deadline 2024-02-10T05:57:32.378782+0000) 2024-02-10T05:57:51.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:51 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:51.175+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:18.080464+0000 front 2024-02-10T05:57:18.080558+0000 (oldest deadline 2024-02-10T05:57:40.380400+0000) 2024-02-10T05:57:51.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:51 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6[51076]: 2024-02-10T05:57:51.175+0000 7f1c969b4700 -1 osd.6 71 heartbeat_check: no reply from 172.21.15.64:6830 osd.3 since back 2024-02-10T05:57:25.681180+0000 front 2024-02-10T05:57:25.681212+0000 (oldest deadline 2024-02-10T05:57:49.781079+0000) 2024-02-10T05:57:51.927 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:51 smithi176 podman[71397]: 2024-02-10 05:57:51.671268676 +0000 UTC m=+5.490428440 container died b5681b7c4beef0f687d9a238dae22a7ed6598fd35bca5b90c0d55dee2cef37b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, ceph=True, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:57:51.928 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:51 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:51.900+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:51.928 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:51 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:51.900+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.372516+0000 front 2024-02-10T05:57:10.372572+0000 (oldest deadline 2024-02-10T05:57:36.272344+0000) 2024-02-10T05:57:51.928 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:51 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:51.900+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:16.272859+0000 front 2024-02-10T05:57:16.272813+0000 (oldest deadline 2024-02-10T05:57:42.172672+0000) 2024-02-10T05:57:51.928 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:51 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:51.900+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6830 osd.3 since back 2024-02-10T05:57:25.073701+0000 front 2024-02-10T05:57:25.073739+0000 (oldest deadline 2024-02-10T05:57:48.573659+0000) 2024-02-10T05:57:51.928 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:51 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:51.900+0000 7f8a3d42a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24416.0:124 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-02-10T05:57:52.238 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:52 smithi176 podman[71397]: 2024-02-10 05:57:52.035220814 +0000 UTC m=+5.854380586 container cleanup b5681b7c4beef0f687d9a238dae22a7ed6598fd35bca5b90c0d55dee2cef37b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-02-10T05:57:52.238 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:52 smithi176 bash[71397]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6 2024-02-10T05:57:52.661 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:52 smithi176 podman[71415]: 2024-02-10 05:57:52.319527789 +0000 UTC m=+0.640873679 container remove b5681b7c4beef0f687d9a238dae22a7ed6598fd35bca5b90c0d55dee2cef37b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_CLEAN=True) 2024-02-10T05:57:52.988 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:52 smithi176 podman[71508]: 2024-02-10 05:57:52.662231514 +0000 UTC m=+0.113595678 container create 9ec5de9682222a6ea3d4de0aae22d88e95384863738683eb1752ecf2a7e5e9b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2) 2024-02-10T05:57:52.988 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:52 smithi176 podman[71508]: 2024-02-10 05:57:52.566539009 +0000 UTC m=+0.017903177 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:57:52.989 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:52 smithi176 podman[71508]: 2024-02-10 05:57:52.889740039 +0000 UTC m=+0.341104212 container init 9ec5de9682222a6ea3d4de0aae22d88e95384863738683eb1752ecf2a7e5e9b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=main-290212a) 2024-02-10T05:57:52.989 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:52 smithi176 podman[71508]: 2024-02-10 05:57:52.893174632 +0000 UTC m=+0.344538800 container start 9ec5de9682222a6ea3d4de0aae22d88e95384863738683eb1752ecf2a7e5e9b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-02-10T05:57:52.989 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:52 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:52.900+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:52.989 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:52 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:52.900+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.372516+0000 front 2024-02-10T05:57:10.372572+0000 (oldest deadline 2024-02-10T05:57:36.272344+0000) 2024-02-10T05:57:52.989 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:52 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:52.900+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:16.272859+0000 front 2024-02-10T05:57:16.272813+0000 (oldest deadline 2024-02-10T05:57:42.172672+0000) 2024-02-10T05:57:52.989 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:52 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:52.900+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6830 osd.3 since back 2024-02-10T05:57:25.073701+0000 front 2024-02-10T05:57:25.073739+0000 (oldest deadline 2024-02-10T05:57:48.573659+0000) 2024-02-10T05:57:52.989 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:52 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:52.900+0000 7f8a3d42a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24416.0:124 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-02-10T05:57:53.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:52 smithi176 podman[71508]: 2024-02-10 05:57:52.989308859 +0000 UTC m=+0.440673032 container attach 9ec5de9682222a6ea3d4de0aae22d88e95384863738683eb1752ecf2a7e5e9b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:57:53.488 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:53 smithi176 podman[71508]: 2024-02-10 05:57:53.192642732 +0000 UTC m=+0.644006897 container died 9ec5de9682222a6ea3d4de0aae22d88e95384863738683eb1752ecf2a7e5e9b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131) 2024-02-10T05:57:53.751 INFO:journalctl@ceph.osd.6.smithi176.stdout:Feb 10 05:57:53 smithi176 podman[71534]: 2024-02-10 05:57:53.734269947 +0000 UTC m=+0.536628719 container remove 9ec5de9682222a6ea3d4de0aae22d88e95384863738683eb1752ecf2a7e5e9b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-02-10T05:57:53.756 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.6.service' 2024-02-10T05:57:53.797 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:57:53.798 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-10T05:57:53.798 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-10T05:57:53.798 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.7 2024-02-10T05:57:54.129 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:53 smithi176 systemd[1]: Stopping Ceph osd.7 for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:57:54.129 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:53 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:53.921+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:54.129 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:53 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:53.921+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.372516+0000 front 2024-02-10T05:57:10.372572+0000 (oldest deadline 2024-02-10T05:57:36.272344+0000) 2024-02-10T05:57:54.129 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:53 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:53.921+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:16.272859+0000 front 2024-02-10T05:57:16.272813+0000 (oldest deadline 2024-02-10T05:57:42.172672+0000) 2024-02-10T05:57:54.129 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:53 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:53.921+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6830 osd.3 since back 2024-02-10T05:57:25.073701+0000 front 2024-02-10T05:57:25.073739+0000 (oldest deadline 2024-02-10T05:57:48.573659+0000) 2024-02-10T05:57:54.129 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:53 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:53.921+0000 7f8a3d42a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24416.0:124 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-02-10T05:57:54.130 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:54 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:54.128+0000 7f8a44e3e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:57:54.130 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:54 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:54.128+0000 7f8a44e3e700 -1 osd.7 71 *** Got signal Terminated *** 2024-02-10T05:57:54.130 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:54 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:54.128+0000 7f8a44e3e700 -1 osd.7 71 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:57:55.239 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:54 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:54.877+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:55.239 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:54 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:54.877+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.372516+0000 front 2024-02-10T05:57:10.372572+0000 (oldest deadline 2024-02-10T05:57:36.272344+0000) 2024-02-10T05:57:55.239 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:54 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:54.877+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:16.272859+0000 front 2024-02-10T05:57:16.272813+0000 (oldest deadline 2024-02-10T05:57:42.172672+0000) 2024-02-10T05:57:55.239 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:54 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:54.877+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6830 osd.3 since back 2024-02-10T05:57:25.073701+0000 front 2024-02-10T05:57:25.073739+0000 (oldest deadline 2024-02-10T05:57:48.573659+0000) 2024-02-10T05:57:55.239 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:54 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:54.877+0000 7f8a3d42a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24416.0:124 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-02-10T05:57:55.988 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:55 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:55.863+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:55.988 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:55 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:55.863+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.372516+0000 front 2024-02-10T05:57:10.372572+0000 (oldest deadline 2024-02-10T05:57:36.272344+0000) 2024-02-10T05:57:55.988 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:55 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:55.863+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:16.272859+0000 front 2024-02-10T05:57:16.272813+0000 (oldest deadline 2024-02-10T05:57:42.172672+0000) 2024-02-10T05:57:55.988 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:55 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:55.863+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6830 osd.3 since back 2024-02-10T05:57:25.073701+0000 front 2024-02-10T05:57:25.073739+0000 (oldest deadline 2024-02-10T05:57:48.573659+0000) 2024-02-10T05:57:55.988 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:55 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:55.863+0000 7f8a3d42a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24416.0:124 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-02-10T05:57:57.239 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:56 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:56.899+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:57.239 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:56 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:56.899+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.372516+0000 front 2024-02-10T05:57:10.372572+0000 (oldest deadline 2024-02-10T05:57:36.272344+0000) 2024-02-10T05:57:57.239 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:56 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:56.899+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:16.272859+0000 front 2024-02-10T05:57:16.272813+0000 (oldest deadline 2024-02-10T05:57:42.172672+0000) 2024-02-10T05:57:57.239 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:56 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:56.899+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6830 osd.3 since back 2024-02-10T05:57:25.073701+0000 front 2024-02-10T05:57:25.073739+0000 (oldest deadline 2024-02-10T05:57:48.573659+0000) 2024-02-10T05:57:57.239 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:56 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:56.899+0000 7f8a3d42a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24416.0:124 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-02-10T05:57:58.238 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:57 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:57.865+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:58.238 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:57 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:57.865+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.372516+0000 front 2024-02-10T05:57:10.372572+0000 (oldest deadline 2024-02-10T05:57:36.272344+0000) 2024-02-10T05:57:58.238 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:57 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:57.865+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:16.272859+0000 front 2024-02-10T05:57:16.272813+0000 (oldest deadline 2024-02-10T05:57:42.172672+0000) 2024-02-10T05:57:58.238 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:57 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:57.865+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6830 osd.3 since back 2024-02-10T05:57:25.073701+0000 front 2024-02-10T05:57:25.073739+0000 (oldest deadline 2024-02-10T05:57:48.573659+0000) 2024-02-10T05:57:58.238 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:57 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:57.866+0000 7f8a3d42a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24416.0:124 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-02-10T05:57:59.238 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:58 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:58.878+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2024-02-10T05:57:02.271177+0000 front 2024-02-10T05:57:02.271267+0000 (oldest deadline 2024-02-10T05:57:23.971044+0000) 2024-02-10T05:57:59.306 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:58 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:58.878+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6814 osd.1 since back 2024-02-10T05:57:10.372516+0000 front 2024-02-10T05:57:10.372572+0000 (oldest deadline 2024-02-10T05:57:36.272344+0000) 2024-02-10T05:57:59.306 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:58 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:58.878+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6822 osd.2 since back 2024-02-10T05:57:16.272859+0000 front 2024-02-10T05:57:16.272813+0000 (oldest deadline 2024-02-10T05:57:42.172672+0000) 2024-02-10T05:57:59.306 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:58 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:58.878+0000 7f8a3d42a700 -1 osd.7 71 heartbeat_check: no reply from 172.21.15.64:6830 osd.3 since back 2024-02-10T05:57:25.073701+0000 front 2024-02-10T05:57:25.073739+0000 (oldest deadline 2024-02-10T05:57:48.573659+0000) 2024-02-10T05:57:59.306 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:58 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7[59107]: 2024-02-10T05:57:58.879+0000 7f8a3d42a700 -1 osd.7 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24416.0:124 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71) 2024-02-10T05:57:59.828 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:59 smithi176 podman[71616]: 2024-02-10 05:57:59.536502596 +0000 UTC m=+5.515945832 container died 9a0aa9ba55984ca9451604951fe273ff4df1593ff41414458c968c571158c41c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:57:59.828 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:59 smithi176 podman[71616]: 2024-02-10 05:57:59.828373287 +0000 UTC m=+5.807816529 container cleanup 9a0aa9ba55984ca9451604951fe273ff4df1593ff41414458c968c571158c41c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T05:58:00.238 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:57:59 smithi176 bash[71616]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7 2024-02-10T05:58:00.238 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:58:00 smithi176 podman[71645]: 2024-02-10 05:58:00.054352035 +0000 UTC m=+0.511656143 container remove 9a0aa9ba55984ca9451604951fe273ff4df1593ff41414458c968c571158c41c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7, ceph=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-02-10T05:58:00.582 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:58:00 smithi176 podman[71736]: 2024-02-10 05:58:00.321945373 +0000 UTC m=+0.094138941 container create 64d08e16b69a227ef645d4faf8f371ad6edbae9a978a5f95aa3483f4d03892d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-02-10T05:58:00.582 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:58:00 smithi176 podman[71736]: 2024-02-10 05:58:00.246150799 +0000 UTC m=+0.018344371 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:58:00.582 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:58:00 smithi176 podman[71736]: 2024-02-10 05:58:00.508024538 +0000 UTC m=+0.280218101 container init 64d08e16b69a227ef645d4faf8f371ad6edbae9a978a5f95aa3483f4d03892d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-290212a) 2024-02-10T05:58:00.582 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:58:00 smithi176 podman[71736]: 2024-02-10 05:58:00.511103312 +0000 UTC m=+0.283296875 container start 64d08e16b69a227ef645d4faf8f371ad6edbae9a978a5f95aa3483f4d03892d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T05:58:00.582 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:58:00 smithi176 podman[71736]: 2024-02-10 05:58:00.582177008 +0000 UTC m=+0.354370584 container attach 64d08e16b69a227ef645d4faf8f371ad6edbae9a978a5f95aa3483f4d03892d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:58:00.988 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:58:00 smithi176 podman[71736]: 2024-02-10 05:58:00.800708665 +0000 UTC m=+0.572902220 container died 64d08e16b69a227ef645d4faf8f371ad6edbae9a978a5f95aa3483f4d03892d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-02-10T05:58:01.332 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.7.service' 2024-02-10T05:58:01.358 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:58:01 smithi176 podman[71763]: 2024-02-10 05:58:01.31844667 +0000 UTC m=+0.512906838 container remove 64d08e16b69a227ef645d4faf8f371ad6edbae9a978a5f95aa3483f4d03892d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T05:58:01.358 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:58:01 smithi176 systemd[1]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.7.service: Deactivated successfully. 2024-02-10T05:58:01.358 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:58:01 smithi176 systemd[1]: Stopped Ceph osd.7 for 00e86dbe-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:58:01.358 INFO:journalctl@ceph.osd.7.smithi176.stdout:Feb 10 05:58:01 smithi176 systemd[1]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@osd.7.service: Consumed 3.249s CPU time. 2024-02-10T05:58:01.374 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:58:01.375 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-10T05:58:01.375 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-02-10T05:58:01.375 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@rgw.foo.a 2024-02-10T05:58:01.697 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Feb 10 05:58:01 smithi064 systemd[1]: Stopping Ceph rgw.foo.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:58:01.976 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Feb 10 05:58:01 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-rgw-foo-a[74505]: 2024-02-10T05:58:01.696+0000 7f0c49b18700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:58:01.976 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Feb 10 05:58:01 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-rgw-foo-a[74505]: 2024-02-10T05:58:01.696+0000 7f0c58389ac0 -1 shutting down 2024-02-10T05:58:12.014 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Feb 10 05:58:11 smithi064 bash[86299]: time="2024-02-10T05:58:11Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-02-10T05:58:12.014 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Feb 10 05:58:11 smithi064 podman[86299]: 2024-02-10 05:58:11.71415096 +0000 UTC m=+10.121166151 container died 760980f4d0054d375eb4232b79fc5b66d11a485c9b443107ab0e21f71742e74b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-rgw-foo-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:58:12.264 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Feb 10 05:58:12 smithi064 podman[86299]: 2024-02-10 05:58:12.263007664 +0000 UTC m=+10.670022855 container cleanup 760980f4d0054d375eb4232b79fc5b66d11a485c9b443107ab0e21f71742e74b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-rgw-foo-a, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2) 2024-02-10T05:58:12.264 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Feb 10 05:58:12 smithi064 bash[86299]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-rgw-foo-a 2024-02-10T05:58:12.740 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Feb 10 05:58:12 smithi064 podman[86325]: 2024-02-10 05:58:12.489206384 +0000 UTC m=+0.767354306 container remove 760980f4d0054d375eb4232b79fc5b66d11a485c9b443107ab0e21f71742e74b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-rgw-foo-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-02-10T05:58:12.740 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Feb 10 05:58:12 smithi064 systemd[1]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-02-10T05:58:12.741 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Feb 10 05:58:12 smithi064 systemd[1]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@rgw.foo.a.service: Failed with result 'exit-code'. 2024-02-10T05:58:12.741 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Feb 10 05:58:12 smithi064 systemd[1]: Stopped Ceph rgw.foo.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:58:12.741 INFO:journalctl@ceph.rgw.foo.a.smithi064.stdout:Feb 10 05:58:12 smithi064 systemd[1]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@rgw.foo.a.service: Consumed 1.620s CPU time. 2024-02-10T05:58:12.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@rgw.foo.a.service' 2024-02-10T05:58:12.783 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:58:12.783 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-02-10T05:58:12.783 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-02-10T05:58:12.784 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@prometheus.a 2024-02-10T05:58:13.086 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Feb 10 05:58:12 smithi176 systemd[1]: Stopping Ceph prometheus.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:58:13.345 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Feb 10 05:58:13 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-prometheus-a[67842]: ts=2024-02-10T05:58:13.087Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-10T05:58:13.345 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Feb 10 05:58:13 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-prometheus-a[67842]: ts=2024-02-10T05:58:13.087Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-02-10T05:58:13.346 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Feb 10 05:58:13 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-prometheus-a[67842]: ts=2024-02-10T05:58:13.087Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-02-10T05:58:13.346 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Feb 10 05:58:13 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-prometheus-a[67842]: ts=2024-02-10T05:58:13.087Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-10T05:58:13.346 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Feb 10 05:58:13 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-prometheus-a[67842]: ts=2024-02-10T05:58:13.087Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-02-10T05:58:13.346 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Feb 10 05:58:13 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-prometheus-a[67842]: ts=2024-02-10T05:58:13.087Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-02-10T05:58:13.346 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Feb 10 05:58:13 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-prometheus-a[67842]: ts=2024-02-10T05:58:13.087Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-02-10T05:58:13.346 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Feb 10 05:58:13 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-prometheus-a[67842]: ts=2024-02-10T05:58:13.087Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-02-10T05:58:13.346 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Feb 10 05:58:13 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-prometheus-a[67842]: ts=2024-02-10T05:58:13.087Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-02-10T05:58:13.346 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Feb 10 05:58:13 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-prometheus-a[67842]: ts=2024-02-10T05:58:13.087Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-02-10T05:58:13.346 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Feb 10 05:58:13 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-prometheus-a[67842]: ts=2024-02-10T05:58:13.087Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-02-10T05:58:13.346 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Feb 10 05:58:13 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-prometheus-a[67842]: ts=2024-02-10T05:58:13.087Z caller=main.go:1147 level=info msg="See you next time!" 2024-02-10T05:58:13.346 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Feb 10 05:58:13 smithi176 podman[71842]: 2024-02-10 05:58:13.097316854 +0000 UTC m=+0.114707274 container died e2ae1af399965b8afc2762a2ed84ec624943f318d0901e0948bfb4618f24e16f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T05:58:13.679 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Feb 10 05:58:13 smithi176 podman[71842]: 2024-02-10 05:58:13.511842017 +0000 UTC m=+0.529232445 container cleanup e2ae1af399965b8afc2762a2ed84ec624943f318d0901e0948bfb4618f24e16f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T05:58:13.679 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Feb 10 05:58:13 smithi176 bash[71842]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-prometheus-a 2024-02-10T05:58:13.892 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@prometheus.a.service' 2024-02-10T05:58:13.929 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:58:13.929 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-02-10T05:58:13.929 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 --force --keep-logs 2024-02-10T05:58:14.125 INFO:teuthology.orchestra.run.smithi064.stdout:Deleting cluster with fsid: 00e86dbe-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:58:16.059 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:58:15 smithi064 systemd[1]: Stopping Ceph alertmanager.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:58:16.318 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:58:16 smithi064 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-alertmanager-a[82864]: ts=2024-02-10T05:58:16.059Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-02-10T05:58:16.318 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:58:16 smithi064 podman[86803]: 2024-02-10 05:58:16.080015668 +0000 UTC m=+0.112973707 container died 0ddfabe698bcf5f4546ba362c4482b91cd36a9cb6c39d5e9c6650481b2bbf3c5 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T05:58:16.626 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:58:16 smithi064 podman[86803]: 2024-02-10 05:58:16.41793922 +0000 UTC m=+0.450897267 container cleanup 0ddfabe698bcf5f4546ba362c4482b91cd36a9cb6c39d5e9c6650481b2bbf3c5 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T05:58:16.626 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:58:16 smithi064 bash[86803]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-alertmanager-a 2024-02-10T05:58:16.966 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:58:16 smithi064 podman[86817]: 2024-02-10 05:58:16.627583059 +0000 UTC m=+0.543731354 container remove 0ddfabe698bcf5f4546ba362c4482b91cd36a9cb6c39d5e9c6650481b2bbf3c5 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T05:58:16.966 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:58:16 smithi064 podman[86817]: 2024-02-10 05:58:16.711438096 +0000 UTC m=+0.627586404 volume remove 6c3e7da3029efa3cc91e36b12af7d16c0f61f96a31d201b0c6aba0107fca4f4d 2024-02-10T05:58:17.376 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:58:17 smithi064 systemd[1]: Stopping Ceph node-exporter.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:58:17.376 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:58:16 smithi064 systemd[1]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-10T05:58:17.376 INFO:journalctl@ceph.alertmanager.a.smithi064.stdout:Feb 10 05:58:16 smithi064 systemd[1]: Stopped Ceph alertmanager.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:58:17.739 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:58:17 smithi064 podman[86930]: 2024-02-10 05:58:17.474804735 +0000 UTC m=+0.144642407 container died fb307795377db7435354bf8f87adbc58022489e20ff007f0081408a95ca52fe3 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T05:58:18.090 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:58:17 smithi064 podman[86930]: 2024-02-10 05:58:17.839312036 +0000 UTC m=+0.509149717 container cleanup fb307795377db7435354bf8f87adbc58022489e20ff007f0081408a95ca52fe3 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T05:58:18.090 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:58:17 smithi064 bash[86930]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a 2024-02-10T05:58:18.376 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:58:18 smithi064 podman[86944]: 2024-02-10 05:58:18.090222599 +0000 UTC m=+0.620223189 container remove fb307795377db7435354bf8f87adbc58022489e20ff007f0081408a95ca52fe3 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T05:58:18.376 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:58:18 smithi064 systemd[1]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-10T05:58:18.376 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:58:18 smithi064 systemd[1]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-10T05:58:18.376 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:58:18 smithi064 systemd[1]: Stopped Ceph node-exporter.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:58:18.376 INFO:journalctl@ceph.node-exporter.a.smithi064.stdout:Feb 10 05:58:18 smithi064 systemd[1]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@node-exporter.a.service: Consumed 1.737s CPU time. 2024-02-10T05:58:18.927 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 --force --keep-logs 2024-02-10T05:58:19.117 INFO:teuthology.orchestra.run.smithi176.stdout:Deleting cluster with fsid: 00e86dbe-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:58:20.690 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:58:20 smithi176 systemd[1]: Stopping Ceph iscsi.iscsi.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:58:20.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:58:20 smithi176 ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a[66249]: debug Shutdown received 2024-02-10T05:58:30.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:58:30 smithi176 bash[72252]: time="2024-02-10T05:58:30Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-02-10T05:58:30.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:58:30 smithi176 podman[72252]: 2024-02-10 05:58:30.714445088 +0000 UTC m=+10.130267433 container died f7e7f53e917618d3625b11bda0634b4bc70acc1abd87090bec727b261a874015 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131) 2024-02-10T05:58:31.317 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:58:31 smithi176 podman[72252]: 2024-02-10 05:58:31.139559176 +0000 UTC m=+10.555381520 container cleanup f7e7f53e917618d3625b11bda0634b4bc70acc1abd87090bec727b261a874015 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:58:31.318 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:58:31 smithi176 bash[72252]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a 2024-02-10T05:58:31.591 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:58:31 smithi176 podman[72266]: 2024-02-10 05:58:31.332243383 +0000 UTC m=+0.618965969 container remove f7e7f53e917618d3625b11bda0634b4bc70acc1abd87090bec727b261a874015 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-iscsi-iscsi-a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-02-10T05:58:31.591 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:58:31 smithi176 systemd[1]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-02-10T05:58:31.945 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:58:31 smithi176 systemd[1]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-02-10T05:58:31.946 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:58:31 smithi176 systemd[1]: Stopped Ceph iscsi.iscsi.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:58:31.946 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Feb 10 05:58:31 smithi176 systemd[1]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@iscsi.iscsi.a.service: Consumed 2.180s CPU time. 2024-02-10T05:58:33.238 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:58:33 smithi176 systemd[1]: Stopping Ceph grafana.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:58:33.632 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:58:33 smithi176 podman[72537]: 2024-02-10 05:58:33.375030943 +0000 UTC m=+0.119736024 container died 1160ad9b2c7638c38d5c09ffd4388bd57c983cb67bcdb40d9a39f7443734b6a8 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, io.openshift.expose-services=, vcs-type=git, vendor=Red Hat, Inc., io.buildah.version=1.32.0, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, release=239.1651231664, architecture=x86_64, build-date=2022-04-29T11:48:11.055897, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8) 2024-02-10T05:58:33.988 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:58:33 smithi176 podman[72537]: 2024-02-10 05:58:33.705831708 +0000 UTC m=+0.450536789 container cleanup 1160ad9b2c7638c38d5c09ffd4388bd57c983cb67bcdb40d9a39f7443734b6a8 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, io.buildah.version=1.32.0, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, vendor=Red Hat, Inc., build-date=2022-04-29T11:48:11.055897, vcs-type=git, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, architecture=x86_64) 2024-02-10T05:58:33.988 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:58:33 smithi176 bash[72537]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a 2024-02-10T05:58:33.988 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:58:33 smithi176 podman[72551]: 2024-02-10 05:58:33.848315004 +0000 UTC m=+0.475335730 container remove 1160ad9b2c7638c38d5c09ffd4388bd57c983cb67bcdb40d9a39f7443734b6a8 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-grafana-a, description=Ceph Grafana Container, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, release=239.1651231664, build-date=2022-04-29T11:48:11.055897, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, name=ubi8, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public) 2024-02-10T05:58:33.988 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:58:33 smithi176 systemd[1]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-02-10T05:58:34.489 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:58:34 smithi176 systemd[1]: Stopping Ceph node-exporter.b for 00e86dbe-c7d8-11ee-95b8-87774f69a715... 2024-02-10T05:58:34.489 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:58:34 smithi176 systemd[1]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T05:58:34.489 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:58:34 smithi176 systemd[1]: Stopped Ceph grafana.a for 00e86dbe-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:58:34.489 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Feb 10 05:58:34 smithi176 systemd[1]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@grafana.a.service: Consumed 3.303s CPU time. 2024-02-10T05:58:34.760 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:58:34 smithi176 podman[72664]: 2024-02-10 05:58:34.503236863 +0000 UTC m=+0.115727604 container died 135900a9fea48ab45ddb6c2dcbffc79e84a1a9fb33ad9c98ac96db37481f9394 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T05:58:35.101 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:58:34 smithi176 podman[72664]: 2024-02-10 05:58:34.859107503 +0000 UTC m=+0.471598244 container cleanup 135900a9fea48ab45ddb6c2dcbffc79e84a1a9fb33ad9c98ac96db37481f9394 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T05:58:35.101 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:58:34 smithi176 bash[72664]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b 2024-02-10T05:58:35.392 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:58:35 smithi176 podman[72678]: 2024-02-10 05:58:35.101804272 +0000 UTC m=+0.603273508 container remove 135900a9fea48ab45ddb6c2dcbffc79e84a1a9fb33ad9c98ac96db37481f9394 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T05:58:35.392 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:58:35 smithi176 systemd[1]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-10T05:58:35.392 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:58:35 smithi176 systemd[1]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-10T05:58:35.392 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:58:35 smithi176 systemd[1]: Stopped Ceph node-exporter.b for 00e86dbe-c7d8-11ee-95b8-87774f69a715. 2024-02-10T05:58:35.392 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Feb 10 05:58:35 smithi176 systemd[1]: ceph-00e86dbe-c7d8-11ee-95b8-87774f69a715@node-exporter.b.service: Consumed 1.658s CPU time. 2024-02-10T05:58:35.880 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:58:35.915 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:58:35.944 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-10T05:58:35.946 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555100/remote/smithi064/crash 2024-02-10T05:58:35.947 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/crash -- . 2024-02-10T05:58:35.998 INFO:teuthology.orchestra.run.smithi064.stderr:tar: /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/crash: Cannot open: No such file or directory 2024-02-10T05:58:35.998 INFO:teuthology.orchestra.run.smithi064.stderr:tar: Error is not recoverable: exiting now 2024-02-10T05:58:36.000 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555100/remote/smithi176/crash 2024-02-10T05:58:36.001 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/crash -- . 2024-02-10T05:58:36.059 INFO:teuthology.orchestra.run.smithi176.stderr:tar: /var/lib/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/crash: Cannot open: No such file or directory 2024-02-10T05:58:36.059 INFO:teuthology.orchestra.run.smithi176.stderr:tar: Error is not recoverable: exiting now 2024-02-10T05:58:36.060 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-10T05:58:36.061 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-10T05:58:36.091 INFO:teuthology.orchestra.run.smithi064.stdout:2024-02-10T05:51:11.758153+0000 mon.a (mon.0) 176 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-02-10T05:58:36.092 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-10T05:58:36.092 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[SEC\]' /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-10T05:58:36.163 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[ERR\]' /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-10T05:58:36.233 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[WRN\]' /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-10T05:58:36.301 INFO:teuthology.orchestra.run.smithi064.stdout:2024-02-10T05:51:11.758153+0000 mon.a (mon.0) 176 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-02-10T05:58:36.301 INFO:tasks.cephadm:Compressing logs... 2024-02-10T05:58:36.301 DEBUG:teuthology.orchestra.run.smithi064:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T05:58:36.343 DEBUG:teuthology.orchestra.run.smithi176:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T05:58:36.370 INFO:teuthology.orchestra.run.smithi064.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-10T05:58:36.370 INFO:teuthology.orchestra.run.smithi064.stderr:: No such file or directory 2024-02-10T05:58:36.370 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-mon.a.log 2024-02-10T05:58:36.370 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-10T05:58:36.371 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-client.rgw.foo.a.log 2024-02-10T05:58:36.371 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-osd.1.log 2024-02-10T05:58:36.371 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-osd.3.log 2024-02-10T05:58:36.371 INFO:teuthology.orchestra.run.smithi064.stderr: 57.7% -- replaced with /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-02-10T05:58:36.371 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-mgr.y.log 2024-02-10T05:58:36.371 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-osd.0.log 2024-02-10T05:58:36.371 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T05:58:36.371 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-osd.0.log: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T05:58:36.378 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph.log 2024-02-10T05:58:36.378 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph.cephadm.log: 93.7% -- replaced with /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T05:58:36.378 INFO:teuthology.orchestra.run.smithi064.stderr:gzip 88.9% -- replaced with /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T05:58:36.378 INFO:teuthology.orchestra.run.smithi064.stderr: -5 --verbose -- /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-osd.2.log 2024-02-10T05:58:36.378 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T05:58:36.378 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-osd.2.log: 91.6% -- replaced with /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-10T05:58:36.379 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-mon.c.log 2024-02-10T05:58:36.381 INFO:teuthology.orchestra.run.smithi176.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-10T05:58:36.381 INFO:teuthology.orchestra.run.smithi176.stderr:: No such file or directory 2024-02-10T05:58:36.381 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-osd.6.log 2024-02-10T05:58:36.382 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-10T05:58:36.382 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-osd.4.log 2024-02-10T05:58:36.382 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-osd.7.log 2024-02-10T05:58:36.382 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-osd.5.log 2024-02-10T05:58:36.382 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-osd.4.log: /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-mgr.x.log 2024-02-10T05:58:36.382 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/tcmu-runner.log 2024-02-10T05:58:36.382 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T05:58:36.382 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/tcmu-runner.log: 90.0% -- replaced with /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T05:58:36.382 INFO:teuthology.orchestra.run.smithi176.stderr: 67.8% -- replaced with /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/tcmu-runner.log.gz 2024-02-10T05:58:36.382 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph.log 2024-02-10T05:58:36.382 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph.cephadm.log: 86.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T05:58:36.382 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T05:58:36.382 INFO:teuthology.orchestra.run.smithi176.stderr: 81.5% -- replaced with /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T05:58:36.383 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-mon.b.log 2024-02-10T05:58:36.383 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-volume.log: 85.6% -- replaced with /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph.log.gz/var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-mon.b.log: 2024-02-10T05:58:36.383 INFO:teuthology.orchestra.run.smithi176.stderr: 90.6% -- replaced with /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-mgr.x.log.gz 2024-02-10T05:58:36.392 INFO:teuthology.orchestra.run.smithi176.stderr: 90.6% -- replaced with /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T05:58:36.394 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-volume.log: /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-mon.c.log: 90.5% -- replaced with /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T05:58:36.508 INFO:teuthology.orchestra.run.smithi064.stderr: 88.9% -- replaced with /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-mgr.y.log.gz 2024-02-10T05:58:36.523 INFO:teuthology.orchestra.run.smithi176.stderr: 93.1% -- replaced with /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-osd.6.log.gz 2024-02-10T05:58:36.526 INFO:teuthology.orchestra.run.smithi176.stderr: 92.1% -- replaced with /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-mon.b.log.gz 2024-02-10T05:58:36.539 INFO:teuthology.orchestra.run.smithi064.stderr: 92.3% -- replaced with /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-mon.c.log.gz 2024-02-10T05:58:36.539 INFO:teuthology.orchestra.run.smithi176.stderr: 93.1% -- replaced with /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-osd.5.log.gz 2024-02-10T05:58:36.550 INFO:teuthology.orchestra.run.smithi176.stderr: 93.2% -- replaced with /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-osd.7.log.gz 2024-02-10T05:58:36.583 INFO:teuthology.orchestra.run.smithi064.stderr: 93.0% -- replaced with /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-osd.2.log.gz 2024-02-10T05:58:36.638 INFO:teuthology.orchestra.run.smithi176.stderr: 93.4% -- replaced with /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-osd.4.log.gz 2024-02-10T05:58:36.640 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-02-10T05:58:36.640 INFO:teuthology.orchestra.run.smithi176.stderr:real 0m0.283s 2024-02-10T05:58:36.640 INFO:teuthology.orchestra.run.smithi176.stderr:user 0m0.928s 2024-02-10T05:58:36.640 INFO:teuthology.orchestra.run.smithi176.stderr:sys 0m0.056s 2024-02-10T05:58:36.658 INFO:teuthology.orchestra.run.smithi064.stderr: 93.4% -- replaced with /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-osd.0.log.gz 2024-02-10T05:58:36.666 INFO:teuthology.orchestra.run.smithi064.stderr: 93.4% -- replaced with /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-osd.3.log.gz 2024-02-10T05:58:36.686 INFO:teuthology.orchestra.run.smithi064.stderr: 93.5% -- replaced with /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-osd.1.log.gz 2024-02-10T05:58:36.708 INFO:teuthology.orchestra.run.smithi064.stderr: 91.1% -- replaced with /var/log/ceph/00e86dbe-c7d8-11ee-95b8-87774f69a715/ceph-mon.a.log.gz 2024-02-10T05:58:36.710 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-02-10T05:58:36.710 INFO:teuthology.orchestra.run.smithi064.stderr:real 0m0.355s 2024-02-10T05:58:36.710 INFO:teuthology.orchestra.run.smithi064.stderr:user 0m1.714s 2024-02-10T05:58:36.710 INFO:teuthology.orchestra.run.smithi064.stderr:sys 0m0.078s 2024-02-10T05:58:36.710 INFO:tasks.cephadm:Archiving logs... 2024-02-10T05:58:36.710 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555100/remote/smithi064/log 2024-02-10T05:58:36.711 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T05:58:37.000 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555100/remote/smithi176/log 2024-02-10T05:58:37.001 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T05:58:37.165 INFO:tasks.cephadm:Removing cluster... 2024-02-10T05:58:37.166 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 --force 2024-02-10T05:58:37.351 INFO:teuthology.orchestra.run.smithi064.stdout:Deleting cluster with fsid: 00e86dbe-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:58:37.472 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 00e86dbe-c7d8-11ee-95b8-87774f69a715 --force 2024-02-10T05:58:37.656 INFO:teuthology.orchestra.run.smithi176.stdout:Deleting cluster with fsid: 00e86dbe-c7d8-11ee-95b8-87774f69a715 2024-02-10T05:58:37.772 INFO:tasks.cephadm:Removing cephadm ... 2024-02-10T05:58:37.772 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T05:58:37.788 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T05:58:37.803 INFO:tasks.cephadm:Teardown complete 2024-02-10T05:58:37.803 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-02-10T05:58:37.813 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi064:/dev/vg_nvme/lv_1... 2024-02-10T05:58:37.813 DEBUG:teuthology.orchestra.run.smithi064:> sudo nvme disconnect -n lv_1 2024-02-10T05:58:37.965 INFO:teuthology.orchestra.run.smithi064.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-02-10T05:58:37.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:58:37.967 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi064:/dev/vg_nvme/lv_2... 2024-02-10T05:58:37.967 DEBUG:teuthology.orchestra.run.smithi064:> sudo nvme disconnect -n lv_2 2024-02-10T05:58:38.109 INFO:teuthology.orchestra.run.smithi064.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-02-10T05:58:38.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:58:38.110 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi064:/dev/vg_nvme/lv_3... 2024-02-10T05:58:38.111 DEBUG:teuthology.orchestra.run.smithi064:> sudo nvme disconnect -n lv_3 2024-02-10T05:58:38.250 INFO:teuthology.orchestra.run.smithi064.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-02-10T05:58:38.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:58:38.252 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi064:/dev/vg_nvme/lv_4... 2024-02-10T05:58:38.252 DEBUG:teuthology.orchestra.run.smithi064:> sudo nvme disconnect -n lv_4 2024-02-10T05:58:38.391 INFO:teuthology.orchestra.run.smithi064.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-02-10T05:58:38.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:58:38.393 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-10T05:58:38.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/scratch_devs 2024-02-10T05:58:38.419 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi176:/dev/vg_nvme/lv_1... 2024-02-10T05:58:38.419 DEBUG:teuthology.orchestra.run.smithi176:> sudo nvme disconnect -n lv_1 2024-02-10T05:58:38.558 INFO:teuthology.orchestra.run.smithi176.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-02-10T05:58:38.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:58:38.560 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi176:/dev/vg_nvme/lv_2... 2024-02-10T05:58:38.560 DEBUG:teuthology.orchestra.run.smithi176:> sudo nvme disconnect -n lv_2 2024-02-10T05:58:38.719 INFO:teuthology.orchestra.run.smithi176.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-02-10T05:58:38.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:58:38.721 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi176:/dev/vg_nvme/lv_3... 2024-02-10T05:58:38.721 DEBUG:teuthology.orchestra.run.smithi176:> sudo nvme disconnect -n lv_3 2024-02-10T05:58:38.845 INFO:teuthology.orchestra.run.smithi176.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-02-10T05:58:38.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:58:38.847 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi176:/dev/vg_nvme/lv_4... 2024-02-10T05:58:38.847 DEBUG:teuthology.orchestra.run.smithi176:> sudo nvme disconnect -n lv_4 2024-02-10T05:58:38.991 INFO:teuthology.orchestra.run.smithi176.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-02-10T05:58:38.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:58:38.993 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-02-10T05:58:38.993 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/scratch_devs 2024-02-10T05:58:39.019 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-10T05:58:39.029 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-10T05:58:39.029 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T05:58:39.032 DEBUG:teuthology.orchestra.run.smithi176:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T05:58:39.044 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-02-10T05:58:39.048 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T05:58:39.048 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-02-10T05:58:39.048 INFO:teuthology.orchestra.run.smithi064.stdout:^* hv01.front.sepia.ceph.com 2 6 377 30 +712us[ +723us] +/- 77ms 2024-02-10T05:58:39.048 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 30 -867us[ -867us] +/- 76ms 2024-02-10T05:58:39.048 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 +279us[ +289us] +/- 104ms 2024-02-10T05:58:39.048 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:58:39.073 INFO:teuthology.orchestra.run.smithi176.stderr:bash: line 1: ntpq: command not found 2024-02-10T05:58:39.077 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T05:58:39.077 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2024-02-10T05:58:39.077 INFO:teuthology.orchestra.run.smithi176.stdout:^* hv01.front.sepia.ceph.com 2 6 377 31 +721us[ +719us] +/- 77ms 2024-02-10T05:58:39.077 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 33 -853us[ -856us] +/- 76ms 2024-02-10T05:58:39.077 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +279us[ +279us] +/- 104ms 2024-02-10T05:58:39.077 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:58:39.078 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-10T05:58:39.087 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-10T05:58:39.088 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-10T05:58:39.096 DEBUG:teuthology.orchestra.run.smithi064:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T05:58:39.172 DEBUG:teuthology.orchestra.run.smithi176:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T05:58:39.241 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T05:58:39.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:58:39.272 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T05:58:39.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:58:39.302 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-10T05:58:39.311 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-10T05:58:39.322 INFO:teuthology.task.internal:Duration was 871.265619 seconds 2024-02-10T05:58:39.323 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-10T05:58:39.331 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-10T05:58:39.331 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T05:58:39.334 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T05:58:39.375 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T05:58:39.384 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T05:58:39.918 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-10T05:58:39.918 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2024-02-10T05:58:39.918 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T05:58:39.977 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi176.front.sepia.ceph.com 2024-02-10T05:58:39.978 DEBUG:teuthology.orchestra.run.smithi176:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T05:58:40.003 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-10T05:58:40.004 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T05:58:40.020 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T05:58:40.072 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-10T05:58:40.072 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T05:58:40.088 DEBUG:teuthology.orchestra.run.smithi176:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T05:58:40.353 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-10T05:58:40.364 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-10T05:58:40.364 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T05:58:40.390 DEBUG:teuthology.orchestra.run.smithi176:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T05:58:40.418 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-10T05:58:40.428 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T05:58:40.433 DEBUG:teuthology.orchestra.run.smithi176:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T05:58:40.456 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2024-02-10T05:58:40.483 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = core 2024-02-10T05:58:40.507 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T05:58:40.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:58:40.546 DEBUG:teuthology.orchestra.run.smithi176:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T05:58:40.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:58:40.562 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-10T05:58:40.572 INFO:teuthology.task.internal:Transferring archived files... 2024-02-10T05:58:40.573 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555100/remote/smithi064 2024-02-10T05:58:40.573 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T05:58:40.627 DEBUG:teuthology.misc:Transferring archived files from smithi176:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555100/remote/smithi176 2024-02-10T05:58:40.627 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T05:58:40.665 INFO:teuthology.task.internal:Removing archive directory... 2024-02-10T05:58:40.665 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T05:58:40.667 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T05:58:40.714 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-10T05:58:40.725 INFO:teuthology.task.internal:Not uploading archives. 2024-02-10T05:58:40.725 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-10T05:58:40.734 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-10T05:58:40.734 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T05:58:40.736 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T05:58:40.748 INFO:teuthology.orchestra.run.smithi064.stdout: 265790 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 05:58 /home/ubuntu/cephtest 2024-02-10T05:58:40.768 INFO:teuthology.orchestra.run.smithi176.stdout: 265832 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 05:58 /home/ubuntu/cephtest 2024-02-10T05:58:40.769 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-10T05:58:40.779 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-10T05:58:40.798 INFO:teuthology.nuke:Checking targets against current locks 2024-02-10T05:58:40.825 DEBUG:teuthology.nuke:shortname: smithi064 2024-02-10T05:58:40.825 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T05:58:40.849 DEBUG:teuthology.nuke:shortname: smithi176 2024-02-10T05:58:40.849 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T05:58:40.853 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555100', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:36:10.721307', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC3oADVnAjG9kBHaL+3tviPsiRUn9yOC3HIBlbR+MQTyM6R4VnjyOZCqmh8FQYIbZewyjMrk2exkNpwpA0mXdiQ='} 2024-02-10T05:58:40.877 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555100', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:36:10.719784', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNfbPxrNPOC1vHKDdw9nlAz0VjN2se4rgoRXzZtrlHaH9TPrXjn45WGqndy6jrY117N+i5Wfh1kTVwNbJPw2M90='} 2024-02-10T05:58:40.907 INFO:teuthology.orchestra.console.smithi064:Power off 2024-02-10T05:58:40.908 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T05:58:40.931 DEBUG:teuthology.orchestra.console.smithi064:power off output: Chassis Power Control: Down/Off 2024-02-10T05:58:40.932 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T05:58:40.947 INFO:teuthology.orchestra.console.smithi176:Power off 2024-02-10T05:58:40.947 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T05:58:40.959 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-02-10T05:58:40.971 DEBUG:teuthology.orchestra.console.smithi176:power off output: Chassis Power Control: Down/Off 2024-02-10T05:58:40.972 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T05:58:40.995 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-02-10T05:58:44.960 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T05:58:44.996 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T05:58:45.070 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-02-10T05:58:45.106 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-02-10T05:58:49.072 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T05:58:49.107 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T05:58:49.182 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is off 2024-02-10T05:58:49.215 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is off 2024-02-10T05:58:49.284 INFO:teuthology.orchestra.console.smithi064:Power off completed 2024-02-10T05:58:49.316 INFO:teuthology.orchestra.console.smithi176:Power off completed 2024-02-10T05:58:49.421 INFO:teuthology.run:Summary data: description: orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} duration: 871.2656192779541 failure_reason: '"2024-02-10T05:51:11.758153+0000 mon.a (mon.0) 176 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log' owner: scheduled_yuriw@teuthology success: false 2024-02-10T05:58:49.421 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T05:58:49.500 INFO:teuthology.run:FAIL