2024-03-10T03:30:47.324 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-10T03:30:47.324 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} --archive /home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587949 --owner scheduled_yuriw@teuthology --name yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587949/orig.config.yaml 2024-03-10T03:30:47.351 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-10T03:30:47.445 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587949 branch: wip-yuri4-testing-2024-03-05-0854 description: rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} email: yweinste@redhat.com first_in_suite: false job_id: '7587949' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri4-testing-2024-03-05-0854 ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - MON_DOWN sha1: 831bc188fa69f143bbba69ac76119e784500e8e3 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 831bc188fa69f143bbba69ac76119e784500e8e3 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri4-testing-2024-03-05-0854 sha1: 831bc188fa69f143bbba69ac76119e784500e8e3 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 6371 sha1: 831bc188fa69f143bbba69ac76119e784500e8e3 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2024-03-05-0854 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_831bc188fa69f143bbba69ac76119e784500e8e3/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 831bc188fa69f143bbba69ac76119e784500e8e3 targets: smithi006.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD1F9RkJkfkAKjjgkfEOQ0i3yUMmXuwZ5bqI4CJd1lFmbJo66FSXFjQEGM+LbgP3CNluxm9nXM3YpBtF/CGB85E= smithi071.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAbrs4181qoZ3+51Gog2JhqrEsleVQk5LCcisz7vdQfkUqS16qOD7CFmreJVG0MJkw5/JCOo02pDWdBm3QnHygU= tasks: - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-08_16:20:46 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-10T03:30:47.445 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_831bc188fa69f143bbba69ac76119e784500e8e3/qa; will attempt to use it 2024-03-10T03:30:47.446 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_831bc188fa69f143bbba69ac76119e784500e8e3/qa/tasks 2024-03-10T03:30:47.446 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-10T03:30:47.448 INFO:teuthology.task.internal:Checking packages... 2024-03-10T03:30:47.475 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '831bc188fa69f143bbba69ac76119e784500e8e3' 2024-03-10T03:30:47.475 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-10T03:30:47.475 INFO:teuthology.packaging:ref: None 2024-03-10T03:30:47.475 INFO:teuthology.packaging:tag: None 2024-03-10T03:30:47.475 INFO:teuthology.packaging:branch: wip-yuri4-testing-2024-03-05-0854 2024-03-10T03:30:47.476 INFO:teuthology.packaging:sha1: 831bc188fa69f143bbba69ac76119e784500e8e3 2024-03-10T03:30:47.476 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri4-testing-2024-03-05-0854 2024-03-10T03:30:47.626 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1875.g831bc188 2024-03-10T03:30:47.629 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-10T03:30:47.635 INFO:teuthology.task.internal:no buildpackages task found 2024-03-10T03:30:47.635 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-10T03:30:47.679 INFO:teuthology.task.internal:Saving configuration 2024-03-10T03:30:47.691 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-10T03:30:47.721 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-10T03:30:47.744 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587949', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-10 03:24:59.728999', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD1F9RkJkfkAKjjgkfEOQ0i3yUMmXuwZ5bqI4CJd1lFmbJo66FSXFjQEGM+LbgP3CNluxm9nXM3YpBtF/CGB85E='} 2024-03-10T03:30:47.764 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587949', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-10 03:24:59.727002', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAbrs4181qoZ3+51Gog2JhqrEsleVQk5LCcisz7vdQfkUqS16qOD7CFmreJVG0MJkw5/JCOo02pDWdBm3QnHygU='} 2024-03-10T03:30:47.765 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-10T03:30:47.771 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-10T03:30:47.771 INFO:teuthology.task.internal:roles: ubuntu@smithi071.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-10T03:30:47.771 INFO:teuthology.run_tasks:Running task console_log... 2024-03-10T03:30:47.882 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9b54110160>, signals=[15]) 2024-03-10T03:30:47.882 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-10T03:30:47.888 INFO:teuthology.task.internal:Opening connections... 2024-03-10T03:30:47.888 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi006.front.sepia.ceph.com 2024-03-10T03:30:47.891 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-10T03:30:48.001 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi071.front.sepia.ceph.com 2024-03-10T03:30:48.002 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-10T03:30:48.089 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-10T03:30:48.096 DEBUG:teuthology.orchestra.run.smithi006:> uname -m 2024-03-10T03:30:48.113 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2024-03-10T03:30:48.114 DEBUG:teuthology.orchestra.run.smithi006:> cat /etc/os-release 2024-03-10T03:30:48.170 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="CentOS Stream" 2024-03-10T03:30:48.170 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="9" 2024-03-10T03:30:48.170 INFO:teuthology.orchestra.run.smithi006.stdout:ID="centos" 2024-03-10T03:30:48.170 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE="rhel fedora" 2024-03-10T03:30:48.170 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="9" 2024-03-10T03:30:48.171 INFO:teuthology.orchestra.run.smithi006.stdout:PLATFORM_ID="platform:el9" 2024-03-10T03:30:48.171 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-10T03:30:48.171 INFO:teuthology.orchestra.run.smithi006.stdout:ANSI_COLOR="0;31" 2024-03-10T03:30:48.171 INFO:teuthology.orchestra.run.smithi006.stdout:LOGO="fedora-logo-icon" 2024-03-10T03:30:48.171 INFO:teuthology.orchestra.run.smithi006.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-10T03:30:48.171 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://centos.org/" 2024-03-10T03:30:48.171 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-10T03:30:48.171 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-10T03:30:48.171 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-10T03:30:48.172 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2024-03-10T03:30:48.206 DEBUG:teuthology.orchestra.run.smithi071:> uname -m 2024-03-10T03:30:48.223 INFO:teuthology.orchestra.run.smithi071.stdout:x86_64 2024-03-10T03:30:48.224 DEBUG:teuthology.orchestra.run.smithi071:> cat /etc/os-release 2024-03-10T03:30:48.279 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="CentOS Stream" 2024-03-10T03:30:48.279 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="9" 2024-03-10T03:30:48.279 INFO:teuthology.orchestra.run.smithi071.stdout:ID="centos" 2024-03-10T03:30:48.279 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE="rhel fedora" 2024-03-10T03:30:48.279 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="9" 2024-03-10T03:30:48.279 INFO:teuthology.orchestra.run.smithi071.stdout:PLATFORM_ID="platform:el9" 2024-03-10T03:30:48.279 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-10T03:30:48.279 INFO:teuthology.orchestra.run.smithi071.stdout:ANSI_COLOR="0;31" 2024-03-10T03:30:48.279 INFO:teuthology.orchestra.run.smithi071.stdout:LOGO="fedora-logo-icon" 2024-03-10T03:30:48.279 INFO:teuthology.orchestra.run.smithi071.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-10T03:30:48.279 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://centos.org/" 2024-03-10T03:30:48.280 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-10T03:30:48.280 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-10T03:30:48.280 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-10T03:30:48.280 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2024-03-10T03:30:48.323 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-10T03:30:48.332 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-10T03:30:48.396 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-10T03:30:48.396 DEBUG:teuthology.orchestra.run.smithi006:> test '!' -e /home/ubuntu/cephtest 2024-03-10T03:30:48.399 DEBUG:teuthology.orchestra.run.smithi071:> test '!' -e /home/ubuntu/cephtest 2024-03-10T03:30:48.414 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-10T03:30:48.437 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-10T03:30:48.437 DEBUG:teuthology.orchestra.run.smithi006:> test -z $(ls -A /var/lib/ceph) 2024-03-10T03:30:48.456 DEBUG:teuthology.orchestra.run.smithi071:> test -z $(ls -A /var/lib/ceph) 2024-03-10T03:30:48.491 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-10T03:30:48.540 INFO:teuthology.run_tasks:Running task kernel... 2024-03-10T03:30:48.563 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-10T03:30:48.563 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-10T03:30:48.563 DEBUG:teuthology.orchestra.run.smithi006:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-10T03:30:48.564 DEBUG:teuthology.orchestra.run.smithi071:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-10T03:30:48.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T03:30:48.579 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-03-10T03:30:48.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T03:30:48.581 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2024-03-10T03:30:48.637 INFO:teuthology.orchestra.run.smithi071.stdout:5.14.0-361.el9.x86_64 2024-03-10T03:30:48.637 INFO:teuthology.task.kernel:Running kernel on smithi071: 5.14.0-361.el9.x86_64 2024-03-10T03:30:48.637 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum install -y kernel 2024-03-10T03:30:48.638 INFO:teuthology.orchestra.run.smithi006.stdout:5.14.0-361.el9.x86_64 2024-03-10T03:30:48.639 INFO:teuthology.task.kernel:Running kernel on smithi006: 5.14.0-361.el9.x86_64 2024-03-10T03:30:48.639 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum install -y kernel 2024-03-10T03:30:50.826 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-10T03:30:50.881 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-10T03:30:52.210 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2024-03-10T03:30:52.340 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-10T03:30:53.139 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-03-10T03:30:56.858 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - BaseOS 1.9 MB/s | 8.0 MB 00:04 2024-03-10T03:30:59.051 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-03-10T03:32:55.628 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - CRB 55 kB/s | 6.1 MB 01:53 2024-03-10T03:32:57.221 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - Extras packages 65 kB/s | 15 kB 00:00 2024-03-10T03:33:08.801 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 143 kB/s | 19 MB 02:14 2024-03-10T03:33:12.876 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-10T03:33:15.435 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 15 kB 00:01 2024-03-10T03:33:17.481 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-10T03:33:22.462 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-10T03:33:24.710 INFO:teuthology.orchestra.run.smithi006.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-10T03:33:24.710 INFO:teuthology.orchestra.run.smithi006.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-10T03:33:24.799 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-10T03:33:24.805 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-10T03:33:24.805 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2024-03-10T03:33:24.805 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-10T03:33:24.805 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-10T03:33:24.805 INFO:teuthology.orchestra.run.smithi006.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-10T03:33:24.805 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-10T03:33:24.805 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-10T03:33:24.805 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-10T03:33:24.805 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-10T03:33:24.805 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:33:24.805 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-10T03:33:24.805 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-10T03:33:24.806 INFO:teuthology.orchestra.run.smithi006.stdout:Install 4 Packages 2024-03-10T03:33:24.806 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:33:24.806 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 100 M 2024-03-10T03:33:24.806 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 123 M 2024-03-10T03:33:24.806 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-10T03:33:25.502 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 748 kB/s | 21 MB 00:28 2024-03-10T03:33:26.705 INFO:teuthology.orchestra.run.smithi006.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 3.7 MB/s | 6.0 MB 00:01 2024-03-10T03:33:27.455 INFO:teuthology.orchestra.run.smithi006.stdout:(2/4): kernel-modules-core-5.14.0-427.el9.x86_6 45 MB/s | 33 MB 00:00 2024-03-10T03:33:29.905 INFO:teuthology.orchestra.run.smithi006.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 8.2 MB/s | 39 MB 00:04 2024-03-10T03:33:30.325 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-10T03:33:30.888 INFO:teuthology.orchestra.run.smithi006.stdout:(4/4): kernel-core-5.14.0-427.el9.x86_64.rpm 3.6 MB/s | 21 MB 00:05 2024-03-10T03:33:30.889 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-10T03:33:30.889 INFO:teuthology.orchestra.run.smithi006.stdout:Total 16 MB/s | 100 MB 00:06 2024-03-10T03:33:31.805 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-10T03:33:32.261 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-10T03:33:32.261 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-10T03:33:32.474 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-10T03:33:32.474 INFO:teuthology.orchestra.run.smithi071.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-10T03:33:32.581 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-03-10T03:33:32.588 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-10T03:33:32.588 INFO:teuthology.orchestra.run.smithi071.stdout: Package Architecture Version Repository Size 2024-03-10T03:33:32.588 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-10T03:33:32.588 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-03-10T03:33:32.588 INFO:teuthology.orchestra.run.smithi071.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-10T03:33:32.588 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-03-10T03:33:32.588 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-10T03:33:32.589 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-10T03:33:32.589 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-10T03:33:32.589 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-10T03:33:32.589 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-03-10T03:33:32.589 INFO:teuthology.orchestra.run.smithi071.stdout:================================================================================ 2024-03-10T03:33:32.589 INFO:teuthology.orchestra.run.smithi071.stdout:Install 4 Packages 2024-03-10T03:33:32.589 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-10T03:33:32.590 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 100 M 2024-03-10T03:33:32.590 INFO:teuthology.orchestra.run.smithi071.stdout:Installed size: 123 M 2024-03-10T03:33:32.590 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-03-10T03:33:33.655 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-10T03:33:33.655 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-10T03:33:36.522 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-10T03:33:37.195 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-10T03:33:37.331 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-10T03:33:40.278 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-10T03:33:40.467 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-10T03:33:45.566 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-10T03:33:45.681 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-10T03:33:50.819 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-10T03:34:21.630 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-10T03:34:21.630 INFO:teuthology.orchestra.run.smithi006.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-10T03:34:21.631 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:34:21.716 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-10T03:34:22.112 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-10T03:34:22.112 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-10T03:34:22.112 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-10T03:34:22.112 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-10T03:34:22.565 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-10T03:34:22.565 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:34:22.565 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-10T03:34:22.565 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-10T03:34:22.565 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-10T03:34:22.565 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-10T03:34:22.565 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-10T03:34:22.565 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:34:22.565 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-10T03:34:23.046 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q kernel | sort -rV | head -n 1 2024-03-10T03:34:23.207 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-10T03:34:23.208 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-10T03:34:23.208 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-10T03:34:23.208 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-10T03:34:23.208 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-10T03:34:23.208 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-10T03:34:23.208 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi006.front.sepia.ceph.com, path=None, version=distro) 2024-03-10T03:34:23.208 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q kernel | sort -rV | head -n 1 2024-03-10T03:34:23.337 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-10T03:34:23.338 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-10T03:34:23.338 DEBUG:teuthology.orchestra.run.smithi006:> sudo rpm -qi grub2-tools 2024-03-10T03:34:23.376 INFO:teuthology.orchestra.run.smithi006.stdout:Name : grub2-tools 2024-03-10T03:34:23.376 INFO:teuthology.orchestra.run.smithi006.stdout:Epoch : 1 2024-03-10T03:34:23.376 INFO:teuthology.orchestra.run.smithi006.stdout:Version : 2.06 2024-03-10T03:34:23.376 INFO:teuthology.orchestra.run.smithi006.stdout:Release : 68.el9 2024-03-10T03:34:23.376 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: x86_64 2024-03-10T03:34:23.376 INFO:teuthology.orchestra.run.smithi006.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-10T03:34:23.376 INFO:teuthology.orchestra.run.smithi006.stdout:Group : Unspecified 2024-03-10T03:34:23.377 INFO:teuthology.orchestra.run.smithi006.stdout:Size : 8264254 2024-03-10T03:34:23.377 INFO:teuthology.orchestra.run.smithi006.stdout:License : GPLv3+ 2024-03-10T03:34:23.377 INFO:teuthology.orchestra.run.smithi006.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-10T03:34:23.377 INFO:teuthology.orchestra.run.smithi006.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-10T03:34:23.377 INFO:teuthology.orchestra.run.smithi006.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-10T03:34:23.377 INFO:teuthology.orchestra.run.smithi006.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-10T03:34:23.377 INFO:teuthology.orchestra.run.smithi006.stdout:Packager : builder@centos.org 2024-03-10T03:34:23.377 INFO:teuthology.orchestra.run.smithi006.stdout:Vendor : CentOS 2024-03-10T03:34:23.377 INFO:teuthology.orchestra.run.smithi006.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-10T03:34:23.377 INFO:teuthology.orchestra.run.smithi006.stdout:Summary : Support tools for GRUB. 2024-03-10T03:34:23.377 INFO:teuthology.orchestra.run.smithi006.stdout:Description : 2024-03-10T03:34:23.377 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:34:23.377 INFO:teuthology.orchestra.run.smithi006.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-10T03:34:23.377 INFO:teuthology.orchestra.run.smithi006.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-10T03:34:23.377 INFO:teuthology.orchestra.run.smithi006.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-10T03:34:23.378 INFO:teuthology.orchestra.run.smithi006.stdout:hardware devices. 2024-03-10T03:34:23.378 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:34:23.378 INFO:teuthology.orchestra.run.smithi006.stdout:This subpackage provides tools for support of all platforms. 2024-03-10T03:34:23.379 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-10T03:34:23.379 INFO:teuthology.task.kernel:Updating grub on smithi006 to boot 5.14.0-427.el9.x86_64 2024-03-10T03:34:23.379 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-10T03:34:24.016 INFO:teuthology.orchestra.run.smithi006.stderr:Generating grub configuration file ... 2024-03-10T03:34:25.465 INFO:teuthology.orchestra.run.smithi006.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-10T03:34:25.509 INFO:teuthology.orchestra.run.smithi006.stderr:done 2024-03-10T03:34:25.512 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-03-10T03:34:25.527 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.PrHsHVMd39 2024-03-10T03:34:25.528 DEBUG:teuthology.orchestra.run.smithi006:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.PrHsHVMd39 2024-03-10T03:34:25.600 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.PrHsHVMd39 2024-03-10T03:34:25.741 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.PrHsHVMd39 is 6KB 2024-03-10T03:34:25.791 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.PrHsHVMd39 2024-03-10T03:34:25.806 DEBUG:teuthology.orchestra.run.smithi006:> sudo /bin/ls /boot/loader/entries || true 2024-03-10T03:34:25.883 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-10T03:34:25.883 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-10T03:34:25.883 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-10T03:34:25.883 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-10T03:34:25.885 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-10T03:34:26.027 DEBUG:teuthology.orchestra.run.smithi006:> sudo shutdown -r now 2024-03-10T03:34:56.029 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-10T03:34:56.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-10T03:34:56.031 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-10T03:35:14.462 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.6 2024-03-10T03:35:23.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-10T03:35:23.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-10T03:35:43.708 INFO:teuthology.orchestra.run.smithi071.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 10 MB/s | 6.0 MB 00:00 2024-03-10T03:35:44.418 INFO:teuthology.orchestra.run.smithi071.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 16 MB/s | 21 MB 00:01 2024-03-10T03:35:45.067 INFO:teuthology.orchestra.run.smithi071.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 25 MB/s | 33 MB 00:01 2024-03-10T03:35:45.642 INFO:teuthology.orchestra.run.smithi071.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 16 MB/s | 39 MB 00:02 2024-03-10T03:35:45.645 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-03-10T03:35:45.645 INFO:teuthology.orchestra.run.smithi071.stdout:Total 766 kB/s | 100 MB 02:13 2024-03-10T03:35:46.528 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-03-10T03:35:46.924 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-03-10T03:35:46.925 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-03-10T03:35:48.260 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-03-10T03:35:48.260 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-03-10T03:35:50.976 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-03-10T03:35:51.604 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-10T03:35:51.744 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-10T03:35:54.515 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-10T03:35:54.701 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-10T03:35:59.429 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-10T03:35:59.653 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-10T03:36:04.432 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-10T03:36:23.467 DEBUG:teuthology.orchestra.remote:timed out 2024-03-10T03:36:33.449 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-10T03:36:33.449 INFO:teuthology.orchestra.run.smithi071.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-10T03:36:33.449 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-10T03:36:33.533 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-10T03:36:33.988 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-10T03:36:33.988 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-10T03:36:33.988 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-10T03:36:33.988 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-10T03:36:34.443 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-10T03:36:34.443 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-10T03:36:34.444 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-03-10T03:36:34.444 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-10T03:36:34.444 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-10T03:36:34.444 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-10T03:36:34.444 INFO:teuthology.orchestra.run.smithi071.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-10T03:36:34.444 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-10T03:36:34.444 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-03-10T03:36:34.962 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2024-03-10T03:36:35.108 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-10T03:36:35.108 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-10T03:36:35.109 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-10T03:36:35.109 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-10T03:36:35.109 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-10T03:36:35.109 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-10T03:36:35.109 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi071.front.sepia.ceph.com, path=None, version=distro) 2024-03-10T03:36:35.109 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q kernel | sort -rV | head -n 1 2024-03-10T03:36:35.228 INFO:teuthology.orchestra.run.smithi071.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-10T03:36:35.229 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-10T03:36:35.230 DEBUG:teuthology.orchestra.run.smithi071:> sudo rpm -qi grub2-tools 2024-03-10T03:36:35.263 INFO:teuthology.orchestra.run.smithi071.stdout:Name : grub2-tools 2024-03-10T03:36:35.263 INFO:teuthology.orchestra.run.smithi071.stdout:Epoch : 1 2024-03-10T03:36:35.263 INFO:teuthology.orchestra.run.smithi071.stdout:Version : 2.06 2024-03-10T03:36:35.263 INFO:teuthology.orchestra.run.smithi071.stdout:Release : 68.el9 2024-03-10T03:36:35.263 INFO:teuthology.orchestra.run.smithi071.stdout:Architecture: x86_64 2024-03-10T03:36:35.263 INFO:teuthology.orchestra.run.smithi071.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-10T03:36:35.263 INFO:teuthology.orchestra.run.smithi071.stdout:Group : Unspecified 2024-03-10T03:36:35.263 INFO:teuthology.orchestra.run.smithi071.stdout:Size : 8264254 2024-03-10T03:36:35.263 INFO:teuthology.orchestra.run.smithi071.stdout:License : GPLv3+ 2024-03-10T03:36:35.264 INFO:teuthology.orchestra.run.smithi071.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-10T03:36:35.264 INFO:teuthology.orchestra.run.smithi071.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-10T03:36:35.264 INFO:teuthology.orchestra.run.smithi071.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-10T03:36:35.264 INFO:teuthology.orchestra.run.smithi071.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-10T03:36:35.265 INFO:teuthology.orchestra.run.smithi071.stdout:Packager : builder@centos.org 2024-03-10T03:36:35.266 INFO:teuthology.orchestra.run.smithi071.stdout:Vendor : CentOS 2024-03-10T03:36:35.266 INFO:teuthology.orchestra.run.smithi071.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-10T03:36:35.266 INFO:teuthology.orchestra.run.smithi071.stdout:Summary : Support tools for GRUB. 2024-03-10T03:36:35.266 INFO:teuthology.orchestra.run.smithi071.stdout:Description : 2024-03-10T03:36:35.266 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-10T03:36:35.266 INFO:teuthology.orchestra.run.smithi071.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-10T03:36:35.266 INFO:teuthology.orchestra.run.smithi071.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-10T03:36:35.266 INFO:teuthology.orchestra.run.smithi071.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-10T03:36:35.266 INFO:teuthology.orchestra.run.smithi071.stdout:hardware devices. 2024-03-10T03:36:35.266 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-10T03:36:35.267 INFO:teuthology.orchestra.run.smithi071.stdout:This subpackage provides tools for support of all platforms. 2024-03-10T03:36:35.268 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-10T03:36:35.268 INFO:teuthology.task.kernel:Updating grub on smithi071 to boot 5.14.0-427.el9.x86_64 2024-03-10T03:36:35.268 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-10T03:36:35.468 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-10T03:36:35.469 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-10T03:36:35.740 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-03-10T03:36:35.857 INFO:teuthology.orchestra.run.smithi071.stderr:Generating grub configuration file ... 2024-03-10T03:36:36.132 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-10T03:36:36.133 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-10T03:36:36.133 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-03-10T03:36:36.148 INFO:teuthology.orchestra.run.smithi006.stdout:5.14.0-427.el9.x86_64 2024-03-10T03:36:36.148 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-10T03:36:36.148 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-10T03:36:36.148 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-10T03:36:37.145 INFO:teuthology.orchestra.run.smithi071.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-10T03:36:37.149 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-10T03:36:37.149 DEBUG:teuthology.orchestra.run.smithi006:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-10T03:36:37.190 INFO:teuthology.orchestra.run.smithi071.stderr:done 2024-03-10T03:36:37.193 DEBUG:teuthology.orchestra.run.smithi071:> mktemp 2024-03-10T03:36:37.207 INFO:teuthology.orchestra.run.smithi071.stdout:/tmp/tmp.uzx4vPWkZm 2024-03-10T03:36:37.208 DEBUG:teuthology.orchestra.run.smithi071:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.uzx4vPWkZm 2024-03-10T03:36:37.216 INFO:teuthology.orchestra.run.smithi006.stdout:ttyS1 2024-03-10T03:36:37.250 DEBUG:teuthology.parallel:result is None 2024-03-10T03:36:37.283 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0666 /tmp/tmp.uzx4vPWkZm 2024-03-10T03:36:37.431 DEBUG:teuthology.orchestra.remote:smithi071:/tmp/tmp.uzx4vPWkZm is 6KB 2024-03-10T03:36:37.479 DEBUG:teuthology.orchestra.run.smithi071:> rm -fr /tmp/tmp.uzx4vPWkZm 2024-03-10T03:36:37.493 DEBUG:teuthology.orchestra.run.smithi071:> sudo /bin/ls /boot/loader/entries || true 2024-03-10T03:36:37.567 INFO:teuthology.orchestra.run.smithi071.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-10T03:36:37.567 INFO:teuthology.orchestra.run.smithi071.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-10T03:36:37.567 INFO:teuthology.orchestra.run.smithi071.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-10T03:36:37.567 INFO:teuthology.orchestra.run.smithi071.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-10T03:36:37.568 DEBUG:teuthology.orchestra.run.smithi071:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-10T03:36:37.734 DEBUG:teuthology.orchestra.run.smithi071:> sudo shutdown -r now 2024-03-10T03:37:07.737 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-10T03:37:07.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-10T03:37:07.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-10T03:37:26.306 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-10T03:37:35.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-10T03:37:35.309 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-10T03:37:38.366 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-10T03:37:50.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-10T03:37:50.368 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-10T03:38:08.802 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.71 2024-03-10T03:38:23.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-10T03:38:23.805 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-10T03:38:24.155 DEBUG:teuthology.orchestra.run.smithi071:> true 2024-03-10T03:38:24.542 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-10T03:38:24.542 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-10T03:38:24.542 DEBUG:teuthology.orchestra.run.smithi071:> uname -r 2024-03-10T03:38:24.556 INFO:teuthology.orchestra.run.smithi071.stdout:5.14.0-427.el9.x86_64 2024-03-10T03:38:24.556 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-10T03:38:24.557 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-10T03:38:24.557 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-10T03:38:25.558 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-10T03:38:25.558 DEBUG:teuthology.orchestra.run.smithi071:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-10T03:38:25.619 INFO:teuthology.orchestra.run.smithi071.stdout:ttyS1 2024-03-10T03:38:25.662 DEBUG:teuthology.parallel:result is None 2024-03-10T03:38:25.663 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-10T03:38:25.670 INFO:teuthology.task.internal:Creating test directory... 2024-03-10T03:38:25.670 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-10T03:38:25.673 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-10T03:38:25.688 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-10T03:38:25.713 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-10T03:38:25.748 INFO:teuthology.task.internal:Creating archive directory... 2024-03-10T03:38:25.748 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-10T03:38:25.751 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-10T03:38:25.780 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-10T03:38:25.787 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-10T03:38:25.787 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-10T03:38:25.818 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-10T03:38:25.846 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-10T03:38:25.859 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-10T03:38:25.864 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-10T03:38:25.879 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-10T03:38:25.880 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-10T03:38:25.888 INFO:teuthology.task.internal:Configuring sudo... 2024-03-10T03:38:25.888 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-10T03:38:25.890 DEBUG:teuthology.orchestra.run.smithi071:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-10T03:38:25.926 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-10T03:38:25.937 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-10T03:38:25.937 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-10T03:38:25.961 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-10T03:38:25.977 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-10T03:38:26.060 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-10T03:38:26.107 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-10T03:38:26.187 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-10T03:38:26.235 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-10T03:38:26.235 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-10T03:38:26.304 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-10T03:38:26.341 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-10T03:38:26.392 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-10T03:38:26.465 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-10T03:38:26.520 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-10T03:38:26.520 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-10T03:38:26.584 DEBUG:teuthology.orchestra.run.smithi006:> sudo service rsyslog restart 2024-03-10T03:38:26.587 DEBUG:teuthology.orchestra.run.smithi071:> sudo service rsyslog restart 2024-03-10T03:38:26.641 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-10T03:38:26.668 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-10T03:38:27.124 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-10T03:38:27.132 INFO:teuthology.task.internal:Starting timer... 2024-03-10T03:38:27.132 INFO:teuthology.run_tasks:Running task pcp... 2024-03-10T03:38:27.176 INFO:teuthology.run_tasks:Running task selinux... 2024-03-10T03:38:27.216 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-10T03:38:27.217 DEBUG:teuthology.orchestra.run.smithi006:> sudo service auditd rotate 2024-03-10T03:38:27.276 INFO:teuthology.orchestra.run.smithi006.stdout:Rotating logs: 2024-03-10T03:38:27.277 DEBUG:teuthology.orchestra.run.smithi071:> sudo service auditd rotate 2024-03-10T03:38:27.340 INFO:teuthology.orchestra.run.smithi071.stdout:Rotating logs: 2024-03-10T03:38:27.341 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-10T03:38:27.341 DEBUG:teuthology.orchestra.run.smithi006:> /usr/sbin/getenforce 2024-03-10T03:38:27.366 INFO:teuthology.orchestra.run.smithi006.stdout:Permissive 2024-03-10T03:38:27.367 DEBUG:teuthology.orchestra.run.smithi071:> /usr/sbin/getenforce 2024-03-10T03:38:27.405 INFO:teuthology.orchestra.run.smithi071.stdout:Permissive 2024-03-10T03:38:27.405 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi006.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi071.front.sepia.ceph.com': 'permissive'} 2024-03-10T03:38:27.405 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-10T03:38:27.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T03:38:27.437 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-10T03:38:27.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T03:38:27.492 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-10T03:38:27.492 DEBUG:teuthology.orchestra.run.smithi006:> sudo /usr/sbin/setenforce permissive 2024-03-10T03:38:27.526 DEBUG:teuthology.orchestra.run.smithi071:> sudo /usr/sbin/setenforce permissive 2024-03-10T03:38:27.791 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-10T03:38:27.801 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-10T03:38:27.809 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-10T03:38:27.936 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-10T03:38:27.957 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-10T03:38:27.958 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi006.front.sepia.ceph.com,smithi071.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-10T03:42:40.859 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi006.front.sepia.ceph.com'), Remote(name='ubuntu@smithi071.front.sepia.ceph.com')] 2024-03-10T03:42:40.861 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-10T03:42:40.862 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-10T03:42:40.953 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-03-10T03:42:41.030 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-10T03:42:41.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-10T03:42:41.030 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-10T03:42:41.117 DEBUG:teuthology.orchestra.run.smithi071:> true 2024-03-10T03:42:41.151 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-03-10T03:42:41.151 INFO:teuthology.run_tasks:Running task clock... 2024-03-10T03:42:41.161 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-10T03:42:41.162 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-10T03:42:41.162 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-10T03:42:41.164 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-10T03:42:41.164 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-10T03:42:41.193 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-10T03:42:41.210 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-10T03:42:41.223 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-10T03:42:41.239 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: ntpd: command not found 2024-03-10T03:42:41.239 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-10T03:42:41.253 INFO:teuthology.orchestra.run.smithi006.stdout:506 Cannot talk to daemon 2024-03-10T03:42:41.268 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: ntpd: command not found 2024-03-10T03:42:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-10T03:42:41.282 INFO:teuthology.orchestra.run.smithi071.stdout:506 Cannot talk to daemon 2024-03-10T03:42:41.285 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-10T03:42:41.298 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-10T03:42:41.314 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-10T03:42:41.338 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2024-03-10T03:42:41.341 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-10T03:42:41.342 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2024-03-10T03:42:41.342 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-10T03:42:41.342 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-10T03:42:41.342 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-10T03:42:41.342 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-10T03:42:41.368 INFO:teuthology.orchestra.run.smithi071.stderr:bash: line 1: ntpq: command not found 2024-03-10T03:42:41.371 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-10T03:42:41.371 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2024-03-10T03:42:41.371 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-10T03:42:41.371 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-10T03:42:41.371 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-10T03:42:41.371 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-10T03:42:41.372 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-03-10T03:42:41.384 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-03-10T03:42:41.385 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-10T03:42:41.385 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2024-03-10T03:42:41.400 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-10T03:42:41.400 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_1 2024-03-10T03:42:41.454 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-10T03:42:41.454 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-10T03:42:41.454 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-10T03:42:41.454 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-10T03:42:41.454 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-10T03:42:41.454 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-10 03:42:39.862367608 +0000 2024-03-10T03:42:41.454 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-10 03:42:18.758566025 +0000 2024-03-10T03:42:41.454 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-10 03:42:18.758566025 +0000 2024-03-10T03:42:41.454 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-10 03:42:18.758566025 +0000 2024-03-10T03:42:41.455 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-10T03:42:41.519 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-10T03:42:41.519 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-10T03:42:41.519 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000312586 s, 1.6 MB/s 2024-03-10T03:42:41.521 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-10T03:42:41.576 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_2 2024-03-10T03:42:41.630 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-10T03:42:41.630 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-10T03:42:41.630 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-10T03:42:41.630 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-10T03:42:41.630 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-10T03:42:41.630 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-10 03:42:39.862367608 +0000 2024-03-10T03:42:41.630 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-10 03:42:19.101562808 +0000 2024-03-10T03:42:41.630 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-10 03:42:19.101562808 +0000 2024-03-10T03:42:41.630 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-10 03:42:19.101562808 +0000 2024-03-10T03:42:41.630 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-10T03:42:41.694 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-10T03:42:41.695 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-10T03:42:41.695 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000173377 s, 3.0 MB/s 2024-03-10T03:42:41.696 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-10T03:42:41.751 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_3 2024-03-10T03:42:41.805 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-10T03:42:41.805 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-10T03:42:41.805 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-10T03:42:41.805 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-10T03:42:41.805 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-10T03:42:41.805 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-10 03:42:39.862367608 +0000 2024-03-10T03:42:41.805 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-10 03:42:19.432559704 +0000 2024-03-10T03:42:41.805 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-10 03:42:19.432559704 +0000 2024-03-10T03:42:41.806 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-10 03:42:19.432559704 +0000 2024-03-10T03:42:41.806 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-10T03:42:41.870 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-10T03:42:41.870 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-10T03:42:41.870 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000174969 s, 2.9 MB/s 2024-03-10T03:42:41.871 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-10T03:42:41.926 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_4 2024-03-10T03:42:41.981 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-10T03:42:41.981 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-10T03:42:41.981 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-10T03:42:41.981 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-10T03:42:41.981 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-10T03:42:41.981 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-10 03:42:39.862367608 +0000 2024-03-10T03:42:41.981 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-10 03:42:19.780556439 +0000 2024-03-10T03:42:41.981 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-10 03:42:19.780556439 +0000 2024-03-10T03:42:41.981 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-10 03:42:19.780556439 +0000 2024-03-10T03:42:41.982 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-10T03:42:42.045 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-10T03:42:42.046 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-10T03:42:42.046 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000173474 s, 3.0 MB/s 2024-03-10T03:42:42.047 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-10T03:42:42.102 DEBUG:teuthology.orchestra.run.smithi006:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-10T03:42:42.246 INFO:teuthology.orchestra.run.smithi006.stdout:loop 2024-03-10T03:42:42.248 INFO:tasks.nvme_loop:Connecting nvme_loop smithi006:/dev/vg_nvme/lv_1... 2024-03-10T03:42:42.248 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-10T03:42:42.285 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-10T03:42:42.322 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/vg_nvme/lv_11 2024-03-10T03:42:42.351 INFO:tasks.nvme_loop:Connecting nvme_loop smithi006:/dev/vg_nvme/lv_2... 2024-03-10T03:42:42.351 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-10T03:42:42.429 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-10T03:42:42.466 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/vg_nvme/lv_21 2024-03-10T03:42:42.494 INFO:tasks.nvme_loop:Connecting nvme_loop smithi006:/dev/vg_nvme/lv_3... 2024-03-10T03:42:42.494 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-10T03:42:42.572 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-10T03:42:42.609 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/vg_nvme/lv_31 2024-03-10T03:42:42.637 INFO:tasks.nvme_loop:Connecting nvme_loop smithi006:/dev/vg_nvme/lv_4... 2024-03-10T03:42:42.638 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-10T03:42:42.715 INFO:teuthology.orchestra.run.smithi006.stdout:1 2024-03-10T03:42:42.754 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/vg_nvme/lv_41 2024-03-10T03:42:42.780 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-10T03:42:42.780 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2024-03-10T03:42:42.834 DEBUG:teuthology.orchestra.run.smithi006:> sudo nvme list 2024-03-10T03:42:42.900 INFO:teuthology.orchestra.run.smithi006.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-10T03:42:42.900 INFO:teuthology.orchestra.run.smithi006.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-10T03:42:42.900 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/nvme0n1 CVFT5330000H400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-10T03:42:42.900 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/nvme1n1 9318b97d0f4313945687 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-10T03:42:42.900 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/nvme2n1 ef6dd47f3d99e7414b53 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-10T03:42:42.900 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/nvme3n1 0f26ccc1b1e21586407e Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-10T03:42:42.900 INFO:teuthology.orchestra.run.smithi006.stdout:/dev/nvme4n1 244524db6efc1c3ab0d1 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-10T03:42:42.901 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-10T03:42:42.901 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-10T03:42:42.901 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/scratch_devs 2024-03-10T03:42:42.967 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-10T03:42:42.967 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2024-03-10T03:42:42.986 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-10T03:42:42.986 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_1 2024-03-10T03:42:43.043 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-10T03:42:43.043 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-10T03:42:43.043 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-10T03:42:43.043 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-10T03:42:43.043 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-03-10T03:42:43.044 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-10 03:42:39.858660028 +0000 2024-03-10T03:42:43.044 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-10 03:42:14.777094940 +0000 2024-03-10T03:42:43.044 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-10 03:42:14.777094940 +0000 2024-03-10T03:42:43.044 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-10 03:42:14.777094940 +0000 2024-03-10T03:42:43.044 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-10T03:42:43.114 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-10T03:42:43.114 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-10T03:42:43.114 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000220656 s, 2.3 MB/s 2024-03-10T03:42:43.115 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-10T03:42:43.174 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_2 2024-03-10T03:42:43.230 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-10T03:42:43.231 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-10T03:42:43.231 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-10T03:42:43.231 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-10T03:42:43.231 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-03-10T03:42:43.231 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-10 03:42:39.858660028 +0000 2024-03-10T03:42:43.231 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-10 03:42:15.110089166 +0000 2024-03-10T03:42:43.231 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-10 03:42:15.110089166 +0000 2024-03-10T03:42:43.231 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-10 03:42:15.110089166 +0000 2024-03-10T03:42:43.231 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-10T03:42:43.299 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-10T03:42:43.299 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-10T03:42:43.299 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000186427 s, 2.7 MB/s 2024-03-10T03:42:43.300 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-10T03:42:43.357 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_3 2024-03-10T03:42:43.413 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-10T03:42:43.413 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-10T03:42:43.413 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-10T03:42:43.413 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-10T03:42:43.413 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-03-10T03:42:43.413 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-10 03:42:39.858660028 +0000 2024-03-10T03:42:43.413 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-10 03:42:15.433083565 +0000 2024-03-10T03:42:43.413 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-10 03:42:15.433083565 +0000 2024-03-10T03:42:43.413 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-10 03:42:15.433083565 +0000 2024-03-10T03:42:43.414 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-10T03:42:43.480 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-10T03:42:43.480 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-10T03:42:43.480 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000216286 s, 2.4 MB/s 2024-03-10T03:42:43.481 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-10T03:42:43.538 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_4 2024-03-10T03:42:43.594 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-10T03:42:43.594 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-10T03:42:43.594 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-10T03:42:43.594 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-10T03:42:43.594 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-03-10T03:42:43.594 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-10 03:42:39.858660028 +0000 2024-03-10T03:42:43.594 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-10 03:42:15.932074912 +0000 2024-03-10T03:42:43.594 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-10 03:42:15.932074912 +0000 2024-03-10T03:42:43.594 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-10 03:42:15.932074912 +0000 2024-03-10T03:42:43.595 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-10T03:42:43.660 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-10T03:42:43.660 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-10T03:42:43.660 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000192193 s, 2.7 MB/s 2024-03-10T03:42:43.661 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-10T03:42:43.718 DEBUG:teuthology.orchestra.run.smithi071:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-10T03:42:43.855 INFO:teuthology.orchestra.run.smithi071.stdout:loop 2024-03-10T03:42:43.856 INFO:tasks.nvme_loop:Connecting nvme_loop smithi071:/dev/vg_nvme/lv_1... 2024-03-10T03:42:43.856 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-10T03:42:43.893 INFO:teuthology.orchestra.run.smithi071.stdout:1 2024-03-10T03:42:43.930 INFO:teuthology.orchestra.run.smithi071.stdout:/dev/vg_nvme/lv_11 2024-03-10T03:42:43.960 INFO:tasks.nvme_loop:Connecting nvme_loop smithi071:/dev/vg_nvme/lv_2... 2024-03-10T03:42:43.960 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-10T03:42:44.045 INFO:teuthology.orchestra.run.smithi071.stdout:1 2024-03-10T03:42:44.084 INFO:teuthology.orchestra.run.smithi071.stdout:/dev/vg_nvme/lv_21 2024-03-10T03:42:44.113 INFO:tasks.nvme_loop:Connecting nvme_loop smithi071:/dev/vg_nvme/lv_3... 2024-03-10T03:42:44.113 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-10T03:42:44.198 INFO:teuthology.orchestra.run.smithi071.stdout:1 2024-03-10T03:42:44.236 INFO:teuthology.orchestra.run.smithi071.stdout:/dev/vg_nvme/lv_31 2024-03-10T03:42:44.273 INFO:tasks.nvme_loop:Connecting nvme_loop smithi071:/dev/vg_nvme/lv_4... 2024-03-10T03:42:44.273 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-10T03:42:44.352 INFO:teuthology.orchestra.run.smithi071.stdout:1 2024-03-10T03:42:44.391 INFO:teuthology.orchestra.run.smithi071.stdout:/dev/vg_nvme/lv_41 2024-03-10T03:42:44.421 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-10T03:42:44.421 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2024-03-10T03:42:44.482 DEBUG:teuthology.orchestra.run.smithi071:> sudo nvme list 2024-03-10T03:42:44.548 INFO:teuthology.orchestra.run.smithi071.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-10T03:42:44.548 INFO:teuthology.orchestra.run.smithi071.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-10T03:42:44.549 INFO:teuthology.orchestra.run.smithi071.stdout:/dev/nvme0n1 PHFT62040172400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-10T03:42:44.549 INFO:teuthology.orchestra.run.smithi071.stdout:/dev/nvme1n1 2a52a3098abc918ed187 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-10T03:42:44.549 INFO:teuthology.orchestra.run.smithi071.stdout:/dev/nvme2n1 c35632b0b7a7a8491f5b Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-10T03:42:44.549 INFO:teuthology.orchestra.run.smithi071.stdout:/dev/nvme3n1 744c282c188a184d7f41 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-10T03:42:44.549 INFO:teuthology.orchestra.run.smithi071.stdout:/dev/nvme4n1 db45d12405b7a1353ffb Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-10T03:42:44.550 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-10T03:42:44.550 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-10T03:42:44.550 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/scratch_devs 2024-03-10T03:42:44.619 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-10T03:42:44.718 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'global': {'mon election default strategy': 3}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'MON_DOWN'], 'sha1': '831bc188fa69f143bbba69ac76119e784500e8e3'} 2024-03-10T03:42:44.719 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 2024-03-10T03:42:44.719 INFO:tasks.cephadm:Cluster fsid is 414ba1e0-de90-11ee-95c7-87774f69a715 2024-03-10T03:42:44.719 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-10T03:42:44.720 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.6', 'mon.c': '[v2:172.21.15.6:3301,v1:172.21.15.6:6790]', 'mon.b': '172.21.15.71'} 2024-03-10T03:42:44.720 INFO:tasks.cephadm:First mon is mon.a on smithi006 2024-03-10T03:42:44.720 INFO:tasks.cephadm:First mgr is y 2024-03-10T03:42:44.720 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-10T03:42:44.720 DEBUG:teuthology.orchestra.run.smithi006:> sudo hostname $(hostname -s) 2024-03-10T03:42:44.751 DEBUG:teuthology.orchestra.run.smithi071:> sudo hostname $(hostname -s) 2024-03-10T03:42:44.778 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-10T03:42:44.778 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=831bc188fa69f143bbba69ac76119e784500e8e3 2024-03-10T03:42:44.929 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '831bc188fa69f143bbba69ac76119e784500e8e3', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77345/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1875-g831bc188', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1875.g831bc188'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2024-03-05-0854/831bc188fa69f143bbba69ac76119e784500e8e3/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-06 02:32:08.274555', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-yuri4-testing-2024-03-05-0854', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-yuri4-testing-2024-03-05-0854/831bc188fa69f143bbba69ac76119e784500e8e3/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-10T03:42:45.071 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-yuri4-testing-2024-03-05-0854, sha1 831bc188fa69f143bbba69ac76119e784500e8e3 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=831bc188fa69f143bbba69ac76119e784500e8e3 2024-03-10T03:42:45.073 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-yuri4-testing-2024-03-05-0854/831bc188fa69f143bbba69ac76119e784500e8e3/centos/9/x86_64/flavors/default/cephadm 2024-03-10T03:42:45.073 DEBUG:teuthology.orchestra.run.smithi006:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-yuri4-testing-2024-03-05-0854/831bc188fa69f143bbba69ac76119e784500e8e3/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-10T03:42:45.866 INFO:teuthology.orchestra.run.smithi006.stdout:-rw-r--r--. 1 ubuntu ubuntu 767443 Mar 10 03:42 /home/ubuntu/cephtest/cephadm 2024-03-10T03:42:45.866 DEBUG:teuthology.orchestra.run.smithi071:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-yuri4-testing-2024-03-05-0854/831bc188fa69f143bbba69ac76119e784500e8e3/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-10T03:42:46.623 INFO:teuthology.orchestra.run.smithi071.stdout:-rw-r--r--. 1 ubuntu ubuntu 767443 Mar 10 03:42 /home/ubuntu/cephtest/cephadm 2024-03-10T03:42:46.623 DEBUG:teuthology.orchestra.run.smithi006:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-10T03:42:46.641 DEBUG:teuthology.orchestra.run.smithi071:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-10T03:42:46.671 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 on all hosts... 2024-03-10T03:42:46.671 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 pull 2024-03-10T03:42:46.683 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 pull 2024-03-10T03:42:47.498 INFO:teuthology.orchestra.run.smithi071.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3... 2024-03-10T03:42:47.588 INFO:teuthology.orchestra.run.smithi006.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3... 2024-03-10T03:43:18.633 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-10T03:43:18.634 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_version": "ceph version 19.0.0-1875-g831bc188 (831bc188fa69f143bbba69ac76119e784500e8e3) squid (dev)", 2024-03-10T03:43:18.634 INFO:teuthology.orchestra.run.smithi006.stdout: "image_id": "01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538", 2024-03-10T03:43:18.634 INFO:teuthology.orchestra.run.smithi006.stdout: "repo_digests": [ 2024-03-10T03:43:18.634 INFO:teuthology.orchestra.run.smithi006.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada" 2024-03-10T03:43:18.634 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-10T03:43:18.634 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-10T03:43:20.788 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-03-10T03:43:20.789 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_version": "ceph version 19.0.0-1875-g831bc188 (831bc188fa69f143bbba69ac76119e784500e8e3) squid (dev)", 2024-03-10T03:43:20.789 INFO:teuthology.orchestra.run.smithi071.stdout: "image_id": "01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538", 2024-03-10T03:43:20.789 INFO:teuthology.orchestra.run.smithi071.stdout: "repo_digests": [ 2024-03-10T03:43:20.789 INFO:teuthology.orchestra.run.smithi071.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada" 2024-03-10T03:43:20.789 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-03-10T03:43:20.789 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-03-10T03:43:20.813 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph 2024-03-10T03:43:20.847 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /etc/ceph 2024-03-10T03:43:20.876 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 777 /etc/ceph 2024-03-10T03:43:20.916 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 777 /etc/ceph 2024-03-10T03:43:20.944 INFO:tasks.cephadm:Writing seed config... 2024-03-10T03:43:20.945 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-10T03:43:20.945 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-10T03:43:20.945 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2024-03-10T03:43:20.945 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-03-10T03:43:20.945 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-10T03:43:20.946 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-10T03:43:20.946 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-10T03:43:20.946 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-10T03:43:20.946 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-10T03:43:20.946 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-10T03:43:20.946 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-10T03:43:20.947 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-10T03:43:20.971 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 414ba1e0-de90-11ee-95c7-87774f69a715 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-10T03:43:20.972 DEBUG:teuthology.orchestra.run.smithi006:mon.a> sudo journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mon.a.service 2024-03-10T03:43:21.015 DEBUG:teuthology.orchestra.run.smithi006:mgr.y> sudo journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mgr.y.service 2024-03-10T03:43:21.058 INFO:tasks.cephadm:Bootstrapping... 2024-03-10T03:43:21.058 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 -v bootstrap --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.6 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-10T03:43:21.255 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-10T03:43:21.255 INFO:teuthology.orchestra.run.smithi006.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3', '-v', 'bootstrap', '--fsid', '414ba1e0-de90-11ee-95c7-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.6', '--skip-admin-label'] 2024-03-10T03:43:21.279 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-10T03:43:21.279 INFO:teuthology.orchestra.run.smithi006.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-10T03:43:21.280 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying podman|docker is present... 2024-03-10T03:43:21.302 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-10T03:43:21.302 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying lvm2 is present... 2024-03-10T03:43:21.303 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying time synchronization is in place... 2024-03-10T03:43:21.310 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-10T03:43:21.310 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-10T03:43:21.315 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-10T03:43:21.315 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout inactive 2024-03-10T03:43:21.322 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout enabled 2024-03-10T03:43:21.327 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout active 2024-03-10T03:43:21.327 INFO:teuthology.orchestra.run.smithi006.stdout:Unit chronyd.service is enabled and running 2024-03-10T03:43:21.328 INFO:teuthology.orchestra.run.smithi006.stdout:Repeating the final host check... 2024-03-10T03:43:21.351 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-10T03:43:21.351 INFO:teuthology.orchestra.run.smithi006.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-10T03:43:21.351 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl is present 2024-03-10T03:43:21.351 INFO:teuthology.orchestra.run.smithi006.stdout:lvcreate is present 2024-03-10T03:43:21.357 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-10T03:43:21.357 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-10T03:43:21.363 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-10T03:43:21.363 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout inactive 2024-03-10T03:43:21.369 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout enabled 2024-03-10T03:43:21.375 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout active 2024-03-10T03:43:21.375 INFO:teuthology.orchestra.run.smithi006.stdout:Unit chronyd.service is enabled and running 2024-03-10T03:43:21.375 INFO:teuthology.orchestra.run.smithi006.stdout:Host looks OK 2024-03-10T03:43:21.375 INFO:teuthology.orchestra.run.smithi006.stdout:Cluster fsid: 414ba1e0-de90-11ee-95c7-87774f69a715 2024-03-10T03:43:21.375 INFO:teuthology.orchestra.run.smithi006.stdout:Acquiring lock 140296361275792 on /run/cephadm/414ba1e0-de90-11ee-95c7-87774f69a715.lock 2024-03-10T03:43:21.375 INFO:teuthology.orchestra.run.smithi006.stdout:Lock 140296361275792 acquired on /run/cephadm/414ba1e0-de90-11ee-95c7-87774f69a715.lock 2024-03-10T03:43:21.376 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying IP 172.21.15.6 port 3300 ... 2024-03-10T03:43:21.376 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying IP 172.21.15.6 port 6789 ... 2024-03-10T03:43:21.376 INFO:teuthology.orchestra.run.smithi006.stdout:Base mon IP(s) is [172.21.15.6:3300, 172.21.15.6:6789], mon addrv is [v2:172.21.15.6:3300,v1:172.21.15.6:6789] 2024-03-10T03:43:21.379 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.6 metric 100 2024-03-10T03:43:21.379 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.6 metric 100 2024-03-10T03:43:21.381 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-10T03:43:21.381 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-10T03:43:21.381 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-10T03:43:21.383 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-10T03:43:21.384 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-10T03:43:21.384 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-10T03:43:21.384 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-10T03:43:21.384 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-10T03:43:21.384 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-10T03:43:21.385 INFO:teuthology.orchestra.run.smithi006.stdout:Mon IP `172.21.15.6` is in CIDR network `172.21.0.0/20` 2024-03-10T03:43:21.385 INFO:teuthology.orchestra.run.smithi006.stdout:Mon IP `172.21.15.6` is in CIDR network `172.21.0.0/20` 2024-03-10T03:43:21.385 INFO:teuthology.orchestra.run.smithi006.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-10T03:43:21.385 INFO:teuthology.orchestra.run.smithi006.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-10T03:43:21.385 INFO:teuthology.orchestra.run.smithi006.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3... 2024-03-10T03:43:21.910 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stdout 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 2024-03-10T03:43:21.910 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3... 2024-03-10T03:43:21.911 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stderr Getting image source signatures 2024-03-10T03:43:21.911 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stderr Copying blob sha256:7feca07754707458c3945cf0062cf4dabc512f6d90fe1a9a1370b362b6011124 2024-03-10T03:43:21.911 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stderr Copying blob sha256:5230e21a7b1561343f4233b6671f80ad8914d8a2054fff55ddda8b9974582e35 2024-03-10T03:43:21.911 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stderr Copying config sha256:01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 2024-03-10T03:43:21.911 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-10T03:43:22.757 INFO:teuthology.orchestra.run.smithi006.stdout:ceph: stdout ceph version 19.0.0-1875-g831bc188 (831bc188fa69f143bbba69ac76119e784500e8e3) squid (dev) 2024-03-10T03:43:22.758 INFO:teuthology.orchestra.run.smithi006.stdout:Ceph version: ceph version 19.0.0-1875-g831bc188 (831bc188fa69f143bbba69ac76119e784500e8e3) squid (dev) 2024-03-10T03:43:22.758 INFO:teuthology.orchestra.run.smithi006.stdout:Extracting ceph user uid/gid from container image... 2024-03-10T03:43:23.705 INFO:teuthology.orchestra.run.smithi006.stdout:stat: stdout 167 167 2024-03-10T03:43:23.705 INFO:teuthology.orchestra.run.smithi006.stdout:Creating initial keys... 2024-03-10T03:43:24.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph-authtool: stdout AQBcLO1lZq9dBBAAPjJdQGM87otZ9bqKZtvJ+Q== 2024-03-10T03:43:25.541 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph-authtool: stdout AQBdLO1l3Qk2ABAA0hDUdIK+QpqP/DQOh/IYug== 2024-03-10T03:43:26.379 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph-authtool: stdout AQBdLO1lFkedMBAAusZMYetlandvX6pFX2kTkA== 2024-03-10T03:43:26.380 INFO:teuthology.orchestra.run.smithi006.stdout:Creating initial monmap... 2024-03-10T03:43:27.210 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-10T03:43:27.210 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-10T03:43:27.210 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 414ba1e0-de90-11ee-95c7-87774f69a715 2024-03-10T03:43:27.210 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-10T03:43:27.211 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool for a [v2:172.21.15.6:3300,v1:172.21.15.6:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-10T03:43:27.211 INFO:teuthology.orchestra.run.smithi006.stdout:setting min_mon_release = quincy 2024-03-10T03:43:27.211 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/monmaptool: set fsid to 414ba1e0-de90-11ee-95c7-87774f69a715 2024-03-10T03:43:27.211 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-10T03:43:27.211 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:43:27.211 INFO:teuthology.orchestra.run.smithi006.stdout:Creating mon... 2024-03-10T03:43:28.100 INFO:teuthology.orchestra.run.smithi006.stdout:create mon.a on 2024-03-10T03:43:28.470 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-10T03:43:28.684 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-414ba1e0-de90-11ee-95c7-87774f69a715.target → /etc/systemd/system/ceph-414ba1e0-de90-11ee-95c7-87774f69a715.target. 2024-03-10T03:43:28.685 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-414ba1e0-de90-11ee-95c7-87774f69a715.target → /etc/systemd/system/ceph-414ba1e0-de90-11ee-95c7-87774f69a715.target. 2024-03-10T03:43:28.931 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mon.a 2024-03-10T03:43:28.932 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Failed to reset failed state of unit ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mon.a.service: Unit ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mon.a.service not loaded. 2024-03-10T03:43:29.114 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-414ba1e0-de90-11ee-95c7-87774f69a715.target.wants/ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mon.a.service → /etc/systemd/system/ceph-414ba1e0-de90-11ee-95c7-87774f69a715@.service. 2024-03-10T03:43:29.639 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-10T03:43:29.639 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout disabled 2024-03-10T03:43:29.645 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-10T03:43:29.645 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout inactive 2024-03-10T03:43:29.645 INFO:teuthology.orchestra.run.smithi006.stdout:firewalld.service is not enabled 2024-03-10T03:43:29.646 INFO:teuthology.orchestra.run.smithi006.stdout:Not possible to enable service . firewalld.service is not available 2024-03-10T03:43:29.646 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for mon to start... 2024-03-10T03:43:29.646 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for mon... 2024-03-10T03:43:30.666 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:30 smithi006 ceph-mon[24883]: from='client.? 172.21.15.6:0/1065906506' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-10T03:43:30.669 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout cluster: 2024-03-10T03:43:30.670 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout id: 414ba1e0-de90-11ee-95c7-87774f69a715 2024-03-10T03:43:30.670 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-10T03:43:30.670 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-10T03:43:30.670 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout services: 2024-03-10T03:43:30.670 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.53004s) 2024-03-10T03:43:30.670 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-10T03:43:30.670 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-10T03:43:30.671 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-10T03:43:30.671 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout data: 2024-03-10T03:43:30.671 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-10T03:43:30.671 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-10T03:43:30.671 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-10T03:43:30.671 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout pgs: 2024-03-10T03:43:30.671 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-10T03:43:30.671 INFO:teuthology.orchestra.run.smithi006.stdout:mon is available 2024-03-10T03:43:30.671 INFO:teuthology.orchestra.run.smithi006.stdout:Assimilating anything we can from ceph.conf... 2024-03-10T03:43:31.684 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-10T03:43:31.684 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout [global] 2024-03-10T03:43:31.684 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout fsid = 414ba1e0-de90-11ee-95c7-87774f69a715 2024-03-10T03:43:31.684 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.6:3300,v1:172.21.15.6:6789] 2024-03-10T03:43:31.684 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-10T03:43:31.684 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-10T03:43:31.684 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-10T03:43:31.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-10T03:43:31.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-10T03:43:31.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-10T03:43:31.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = True 2024-03-10T03:43:31.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-10T03:43:31.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-10T03:43:31.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout [osd] 2024-03-10T03:43:31.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-10T03:43:31.685 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-10T03:43:31.685 INFO:teuthology.orchestra.run.smithi006.stdout:Generating new minimal ceph.conf... 2024-03-10T03:43:31.788 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:31 smithi006 ceph-mon[24883]: from='client.? 172.21.15.6:0/982014120' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-10T03:43:31.789 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:31 smithi006 ceph-mon[24883]: from='client.? 172.21.15.6:0/982014120' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-10T03:43:32.680 INFO:teuthology.orchestra.run.smithi006.stdout:Restarting the monitor... 2024-03-10T03:43:32.704 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:32 smithi006 ceph-mon[24883]: from='client.? 172.21.15.6:0/1426679002' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:43:32.976 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:32 smithi006 systemd[1]: Stopping Ceph mon.a for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:43:33.331 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:32 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-a[24879]: 2024-03-10T03:43:32.974+0000 7f9cc8551700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T03:43:33.331 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:32 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-a[24879]: 2024-03-10T03:43:32.974+0000 7f9cc8551700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-10T03:43:33.331 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:33 smithi006 podman[25129]: 2024-03-10 03:43:33.077075385 +0000 UTC m=+0.227689700 container died ff9d8be0c3a2d459ab35f0b97d228aa540dd6a85c53f93f4e14074ed109df55a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-a, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-10T03:43:33.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:33 smithi006 podman[25129]: 2024-03-10 03:43:33.421481347 +0000 UTC m=+0.572095659 container cleanup ff9d8be0c3a2d459ab35f0b97d228aa540dd6a85c53f93f4e14074ed109df55a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-10T03:43:33.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:33 smithi006 bash[25129]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-a 2024-03-10T03:43:34.057 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:33 smithi006 podman[25143]: 2024-03-10 03:43:33.615247446 +0000 UTC m=+0.533365507 container remove ff9d8be0c3a2d459ab35f0b97d228aa540dd6a85c53f93f4e14074ed109df55a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-10T03:43:34.057 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:33 smithi006 systemd[1]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-10T03:43:34.057 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:33 smithi006 systemd[1]: Stopped Ceph mon.a for 414ba1e0-de90-11ee-95c7-87774f69a715. 2024-03-10T03:43:34.057 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:33 smithi006 systemd[1]: Starting Ceph mon.a for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:43:34.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:34 smithi006 podman[25237]: 2024-03-10 03:43:33.959543982 +0000 UTC m=+0.018251231 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 2024-03-10T03:43:35.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:35 smithi006 podman[25237]: 2024-03-10 03:43:35.296246091 +0000 UTC m=+1.354953375 container create 95f492a8209d21bc1066387178a63d18d00bc022f85e2ab0cca33b5359771c1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-10T03:43:37.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 podman[25237]: 2024-03-10 03:43:37.226696138 +0000 UTC m=+3.285403381 container init 95f492a8209d21bc1066387178a63d18d00bc022f85e2ab0cca33b5359771c1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-a, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-10T03:43:37.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 podman[25237]: 2024-03-10 03:43:37.230886405 +0000 UTC m=+3.289593650 container start 95f492a8209d21bc1066387178a63d18d00bc022f85e2ab0cca33b5359771c1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-10T03:43:37.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: set uid:gid to 167:167 (ceph:ceph) 2024-03-10T03:43:37.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: ceph version 19.0.0-1875-g831bc188 (831bc188fa69f143bbba69ac76119e784500e8e3) squid (dev), process ceph-mon, pid 2 2024-03-10T03:43:37.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: pidfile_write: ignore empty --pid-file 2024-03-10T03:43:37.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: load: jerasure load: lrc 2024-03-10T03:43:37.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: RocksDB version: 7.9.2 2024-03-10T03:43:37.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Git sha 0 2024-03-10T03:43:37.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Compile date 2024-03-06 01:10:31 2024-03-10T03:43:37.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: DB SUMMARY 2024-03-10T03:43:37.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: DB Session ID: 7CDV1PJQWVTF6HLA80KB 2024-03-10T03:43:37.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: CURRENT file: CURRENT 2024-03-10T03:43:37.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: IDENTITY file: IDENTITY 2024-03-10T03:43:37.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-10T03:43:37.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-10T03:43:37.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84593 ; 2024-03-10T03:43:37.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.error_if_exists: 0 2024-03-10T03:43:37.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.create_if_missing: 0 2024-03-10T03:43:37.589 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.paranoid_checks: 1 2024-03-10T03:43:37.589 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-10T03:43:37.589 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-10T03:43:37.589 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-10T03:43:37.589 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.env: 0x55febe447b20 2024-03-10T03:43:37.589 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.fs: PosixFileSystem 2024-03-10T03:43:37.589 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.info_log: 0x55fec00958e0 2024-03-10T03:43:37.589 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_file_opening_threads: 16 2024-03-10T03:43:37.589 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.statistics: (nil) 2024-03-10T03:43:37.589 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.use_fsync: 0 2024-03-10T03:43:37.589 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_log_file_size: 0 2024-03-10T03:43:37.589 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-10T03:43:37.590 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-10T03:43:37.590 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.keep_log_file_num: 1000 2024-03-10T03:43:37.590 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.recycle_log_file_num: 0 2024-03-10T03:43:37.590 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.allow_fallocate: 1 2024-03-10T03:43:37.590 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.allow_mmap_reads: 0 2024-03-10T03:43:37.590 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.allow_mmap_writes: 0 2024-03-10T03:43:37.590 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.use_direct_reads: 0 2024-03-10T03:43:37.590 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-10T03:43:37.590 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.create_missing_column_families: 0 2024-03-10T03:43:37.590 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.db_log_dir: 2024-03-10T03:43:37.590 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.wal_dir: 2024-03-10T03:43:37.590 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-10T03:43:37.590 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-10T03:43:37.590 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-10T03:43:37.591 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-10T03:43:37.591 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-10T03:43:37.591 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-10T03:43:37.591 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.advise_random_on_open: 1 2024-03-10T03:43:37.591 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.db_write_buffer_size: 0 2024-03-10T03:43:37.591 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.write_buffer_manager: 0x55fec0069400 2024-03-10T03:43:37.591 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-10T03:43:37.591 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-10T03:43:37.591 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-10T03:43:37.591 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.rate_limiter: (nil) 2024-03-10T03:43:37.591 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-10T03:43:37.591 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.wal_recovery_mode: 2 2024-03-10T03:43:37.591 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.enable_thread_tracking: 0 2024-03-10T03:43:37.591 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.enable_pipelined_write: 0 2024-03-10T03:43:37.592 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.unordered_write: 0 2024-03-10T03:43:37.592 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-10T03:43:37.592 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-10T03:43:37.592 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-10T03:43:37.592 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-10T03:43:37.592 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.row_cache: None 2024-03-10T03:43:37.592 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.wal_filter: None 2024-03-10T03:43:37.593 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-10T03:43:37.593 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.allow_ingest_behind: 0 2024-03-10T03:43:37.593 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.two_write_queues: 0 2024-03-10T03:43:37.593 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.manual_wal_flush: 0 2024-03-10T03:43:37.593 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.wal_compression: 0 2024-03-10T03:43:37.594 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.atomic_flush: 0 2024-03-10T03:43:37.594 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-10T03:43:37.594 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-10T03:43:37.594 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-10T03:43:37.594 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.log_readahead_size: 0 2024-03-10T03:43:37.594 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-10T03:43:37.594 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.best_efforts_recovery: 0 2024-03-10T03:43:37.594 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-10T03:43:37.594 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-10T03:43:37.594 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.allow_data_in_errors: 0 2024-03-10T03:43:37.594 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.db_host_id: __hostname__ 2024-03-10T03:43:37.594 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-10T03:43:37.594 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_background_jobs: 2 2024-03-10T03:43:37.595 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_background_compactions: -1 2024-03-10T03:43:37.595 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_subcompactions: 1 2024-03-10T03:43:37.595 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-10T03:43:37.595 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-10T03:43:37.595 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-10T03:43:37.595 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_total_wal_size: 0 2024-03-10T03:43:37.595 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-10T03:43:37.595 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-10T03:43:37.595 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-10T03:43:37.595 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-10T03:43:37.595 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_open_files: -1 2024-03-10T03:43:37.595 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.bytes_per_sync: 0 2024-03-10T03:43:37.595 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-10T03:43:37.595 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-10T03:43:37.596 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.compaction_readahead_size: 0 2024-03-10T03:43:37.596 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_background_flushes: -1 2024-03-10T03:43:37.596 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Compression algorithms supported: 2024-03-10T03:43:37.596 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-10T03:43:37.596 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: kZSTD supported: 0 2024-03-10T03:43:37.596 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: kXpressCompression supported: 0 2024-03-10T03:43:37.596 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: kLZ4HCCompression supported: 1 2024-03-10T03:43:37.596 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: kZlibCompression supported: 1 2024-03-10T03:43:37.596 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: kSnappyCompression supported: 1 2024-03-10T03:43:37.596 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: kLZ4Compression supported: 1 2024-03-10T03:43:37.596 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: kBZip2Compression supported: 0 2024-03-10T03:43:37.596 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-10T03:43:37.596 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-10T03:43:37.596 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-10T03:43:37.596 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-10T03:43:37.597 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-10T03:43:37.597 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.merge_operator: 2024-03-10T03:43:37.597 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.compaction_filter: None 2024-03-10T03:43:37.597 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.compaction_filter_factory: None 2024-03-10T03:43:37.597 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.sst_partitioner_factory: None 2024-03-10T03:43:37.597 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-10T03:43:37.597 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-10T03:43:37.597 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fec00959e0) 2024-03-10T03:43:37.597 INFO:journalctl@ceph.mon.a.smithi006.stdout: cache_index_and_filter_blocks: 1 2024-03-10T03:43:37.597 INFO:journalctl@ceph.mon.a.smithi006.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-10T03:43:37.597 INFO:journalctl@ceph.mon.a.smithi006.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-10T03:43:37.597 INFO:journalctl@ceph.mon.a.smithi006.stdout: pin_top_level_index_and_filter: 1 2024-03-10T03:43:37.597 INFO:journalctl@ceph.mon.a.smithi006.stdout: index_type: 0 2024-03-10T03:43:37.597 INFO:journalctl@ceph.mon.a.smithi006.stdout: data_block_index_type: 0 2024-03-10T03:43:37.598 INFO:journalctl@ceph.mon.a.smithi006.stdout: index_shortening: 1 2024-03-10T03:43:37.598 INFO:journalctl@ceph.mon.a.smithi006.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-10T03:43:37.598 INFO:journalctl@ceph.mon.a.smithi006.stdout: checksum: 4 2024-03-10T03:43:37.598 INFO:journalctl@ceph.mon.a.smithi006.stdout: no_block_cache: 0 2024-03-10T03:43:37.598 INFO:journalctl@ceph.mon.a.smithi006.stdout: block_cache: 0x55fec009b090 2024-03-10T03:43:37.598 INFO:journalctl@ceph.mon.a.smithi006.stdout: block_cache_name: BinnedLRUCache 2024-03-10T03:43:37.598 INFO:journalctl@ceph.mon.a.smithi006.stdout: block_cache_options: 2024-03-10T03:43:37.598 INFO:journalctl@ceph.mon.a.smithi006.stdout: capacity : 536870912 2024-03-10T03:43:37.598 INFO:journalctl@ceph.mon.a.smithi006.stdout: num_shard_bits : 4 2024-03-10T03:43:37.598 INFO:journalctl@ceph.mon.a.smithi006.stdout: strict_capacity_limit : 0 2024-03-10T03:43:37.598 INFO:journalctl@ceph.mon.a.smithi006.stdout: high_pri_pool_ratio: 0.000 2024-03-10T03:43:37.598 INFO:journalctl@ceph.mon.a.smithi006.stdout: block_cache_compressed: (nil) 2024-03-10T03:43:37.598 INFO:journalctl@ceph.mon.a.smithi006.stdout: persistent_cache: (nil) 2024-03-10T03:43:37.598 INFO:journalctl@ceph.mon.a.smithi006.stdout: block_size: 4096 2024-03-10T03:43:37.599 INFO:journalctl@ceph.mon.a.smithi006.stdout: block_size_deviation: 10 2024-03-10T03:43:37.599 INFO:journalctl@ceph.mon.a.smithi006.stdout: block_restart_interval: 16 2024-03-10T03:43:37.599 INFO:journalctl@ceph.mon.a.smithi006.stdout: index_block_restart_interval: 1 2024-03-10T03:43:37.599 INFO:journalctl@ceph.mon.a.smithi006.stdout: metadata_block_size: 4096 2024-03-10T03:43:37.599 INFO:journalctl@ceph.mon.a.smithi006.stdout: partition_filters: 0 2024-03-10T03:43:37.599 INFO:journalctl@ceph.mon.a.smithi006.stdout: use_delta_encoding: 1 2024-03-10T03:43:37.599 INFO:journalctl@ceph.mon.a.smithi006.stdout: filter_policy: bloomfilter 2024-03-10T03:43:37.599 INFO:journalctl@ceph.mon.a.smithi006.stdout: whole_key_filtering: 1 2024-03-10T03:43:37.599 INFO:journalctl@ceph.mon.a.smithi006.stdout: verify_compression: 0 2024-03-10T03:43:37.599 INFO:journalctl@ceph.mon.a.smithi006.stdout: read_amp_bytes_per_bit: 0 2024-03-10T03:43:37.599 INFO:journalctl@ceph.mon.a.smithi006.stdout: format_version: 5 2024-03-10T03:43:37.599 INFO:journalctl@ceph.mon.a.smithi006.stdout: enable_index_compression: 1 2024-03-10T03:43:37.599 INFO:journalctl@ceph.mon.a.smithi006.stdout: block_align: 0 2024-03-10T03:43:37.599 INFO:journalctl@ceph.mon.a.smithi006.stdout: max_auto_readahead_size: 262144 2024-03-10T03:43:37.600 INFO:journalctl@ceph.mon.a.smithi006.stdout: prepopulate_block_cache: 0 2024-03-10T03:43:37.600 INFO:journalctl@ceph.mon.a.smithi006.stdout: initial_auto_readahead_size: 8192 2024-03-10T03:43:37.600 INFO:journalctl@ceph.mon.a.smithi006.stdout: num_file_reads_for_auto_readahead: 2 2024-03-10T03:43:37.600 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.write_buffer_size: 33554432 2024-03-10T03:43:37.600 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_write_buffer_number: 2 2024-03-10T03:43:37.600 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.compression: NoCompression 2024-03-10T03:43:37.600 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.bottommost_compression: Disabled 2024-03-10T03:43:37.600 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.prefix_extractor: nullptr 2024-03-10T03:43:37.600 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-10T03:43:37.600 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.num_levels: 7 2024-03-10T03:43:37.600 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-10T03:43:37.600 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-10T03:43:37.600 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-10T03:43:37.600 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-10T03:43:37.601 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-10T03:43:37.601 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-10T03:43:37.601 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-10T03:43:37.601 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-10T03:43:37.601 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-10T03:43:37.601 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-10T03:43:37.601 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-10T03:43:37.601 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-10T03:43:37.601 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-10T03:43:37.601 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.compression_opts.level: 32767 2024-03-10T03:43:37.601 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.compression_opts.strategy: 0 2024-03-10T03:43:37.601 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-10T03:43:37.601 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-10T03:43:37.601 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-10T03:43:37.602 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-10T03:43:37.602 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.compression_opts.enabled: false 2024-03-10T03:43:37.602 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-10T03:43:37.602 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-10T03:43:37.602 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-10T03:43:37.602 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-10T03:43:37.602 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.target_file_size_base: 67108864 2024-03-10T03:43:37.602 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-10T03:43:37.602 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-10T03:43:37.602 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-10T03:43:37.602 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-10T03:43:37.602 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-10T03:43:37.602 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-10T03:43:37.603 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-10T03:43:37.603 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-10T03:43:37.603 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-10T03:43:37.603 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-10T03:43:37.603 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-10T03:43:37.603 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-10T03:43:37.603 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-10T03:43:37.603 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-10T03:43:37.603 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.arena_block_size: 1048576 2024-03-10T03:43:37.603 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-10T03:43:37.603 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-10T03:43:37.603 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.disable_auto_compactions: 0 2024-03-10T03:43:37.603 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-10T03:43:37.603 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-10T03:43:37.603 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-10T03:43:37.604 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-10T03:43:37.604 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-10T03:43:37.604 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-10T03:43:37.604 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-10T03:43:37.604 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-10T03:43:37.604 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-10T03:43:37.604 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-10T03:43:37.604 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.table_properties_collectors: 2024-03-10T03:43:37.604 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.inplace_update_support: 0 2024-03-10T03:43:37.604 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-10T03:43:37.604 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-10T03:43:37.604 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-10T03:43:37.604 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-10T03:43:37.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.bloom_locality: 0 2024-03-10T03:43:37.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.max_successive_merges: 0 2024-03-10T03:43:37.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-10T03:43:37.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.paranoid_file_checks: 0 2024-03-10T03:43:37.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.force_consistency_checks: 1 2024-03-10T03:43:37.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.report_bg_io_stats: 0 2024-03-10T03:43:37.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.ttl: 2592000 2024-03-10T03:43:37.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-10T03:43:37.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-10T03:43:37.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-10T03:43:37.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.enable_blob_files: false 2024-03-10T03:43:37.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.min_blob_size: 0 2024-03-10T03:43:37.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.blob_file_size: 268435456 2024-03-10T03:43:37.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-10T03:43:37.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-10T03:43:37.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-10T03:43:37.606 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-10T03:43:37.606 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-10T03:43:37.606 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.blob_file_starting_level: 0 2024-03-10T03:43:37.606 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-10T03:43:37.606 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-10T03:43:37.606 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-10T03:43:37.606 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: df5f3ae4-e3cb-4719-bc84-6e6fcba1a5c3 2024-03-10T03:43:37.606 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710042217264596, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-10T03:43:37.606 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-10T03:43:37.606 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710042217265586, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81218, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 79405, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 73714, "raw_average_value_size": 349, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710042217, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "df5f3ae4-e3cb-4719-bc84-6e6fcba1a5c3", "db_session_id": "7CDV1PJQWVTF6HLA80KB", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-10T03:43:37.606 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710042217265687, "job": 1, "event": "recovery_finished"} 2024-03-10T03:43:37.606 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-10T03:43:37.606 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-10T03:43:37.606 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fec018e000 2024-03-10T03:43:37.607 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: DB pointer 0x55fec0178000 2024-03-10T03:43:37.607 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-10T03:43:37.607 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-10T03:43:37.607 INFO:journalctl@ceph.mon.a.smithi006.stdout: ** DB Stats ** 2024-03-10T03:43:37.607 INFO:journalctl@ceph.mon.a.smithi006.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-10T03:43:37.607 INFO:journalctl@ceph.mon.a.smithi006.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-10T03:43:37.607 INFO:journalctl@ceph.mon.a.smithi006.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-10T03:43:37.607 INFO:journalctl@ceph.mon.a.smithi006.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-10T03:43:37.607 INFO:journalctl@ceph.mon.a.smithi006.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-10T03:43:37.607 INFO:journalctl@ceph.mon.a.smithi006.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-10T03:43:37.608 INFO:journalctl@ceph.mon.a.smithi006.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-10T03:43:37.608 INFO:journalctl@ceph.mon.a.smithi006.stdout: 2024-03-10T03:43:37.608 INFO:journalctl@ceph.mon.a.smithi006.stdout: ** Compaction Stats [default] ** 2024-03-10T03:43:37.608 INFO:journalctl@ceph.mon.a.smithi006.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-10T03:43:37.608 INFO:journalctl@ceph.mon.a.smithi006.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-10T03:43:37.608 INFO:journalctl@ceph.mon.a.smithi006.stdout: L0 2/0 81.15 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 117.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-10T03:43:37.608 INFO:journalctl@ceph.mon.a.smithi006.stdout: Sum 2/0 81.15 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 117.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-10T03:43:37.608 INFO:journalctl@ceph.mon.a.smithi006.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 117.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-10T03:43:37.608 INFO:journalctl@ceph.mon.a.smithi006.stdout: 2024-03-10T03:43:37.608 INFO:journalctl@ceph.mon.a.smithi006.stdout: ** Compaction Stats [default] ** 2024-03-10T03:43:37.608 INFO:journalctl@ceph.mon.a.smithi006.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-10T03:43:37.608 INFO:journalctl@ceph.mon.a.smithi006.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-10T03:43:37.608 INFO:journalctl@ceph.mon.a.smithi006.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 117.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-10T03:43:37.608 INFO:journalctl@ceph.mon.a.smithi006.stdout: 2024-03-10T03:43:37.609 INFO:journalctl@ceph.mon.a.smithi006.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-10T03:43:37.609 INFO:journalctl@ceph.mon.a.smithi006.stdout: 2024-03-10T03:43:37.609 INFO:journalctl@ceph.mon.a.smithi006.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-10T03:43:37.609 INFO:journalctl@ceph.mon.a.smithi006.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-10T03:43:37.609 INFO:journalctl@ceph.mon.a.smithi006.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-10T03:43:37.609 INFO:journalctl@ceph.mon.a.smithi006.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-10T03:43:37.609 INFO:journalctl@ceph.mon.a.smithi006.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-10T03:43:37.609 INFO:journalctl@ceph.mon.a.smithi006.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-10T03:43:37.609 INFO:journalctl@ceph.mon.a.smithi006.stdout: Cumulative compaction: 0.00 GB write, 22.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-10T03:43:37.609 INFO:journalctl@ceph.mon.a.smithi006.stdout: Interval compaction: 0.00 GB write, 22.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-10T03:43:37.609 INFO:journalctl@ceph.mon.a.smithi006.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-10T03:43:37.609 INFO:journalctl@ceph.mon.a.smithi006.stdout: Block cache BinnedLRUCache@0x55fec009b090#2 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-10T03:43:37.609 INFO:journalctl@ceph.mon.a.smithi006.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-03-10T03:43:37.609 INFO:journalctl@ceph.mon.a.smithi006.stdout: 2024-03-10T03:43:37.609 INFO:journalctl@ceph.mon.a.smithi006.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-10T03:43:37.609 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: starting mon.a rank 0 at public addrs [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] at bind addrs [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 414ba1e0-de90-11ee-95c7-87774f69a715 2024-03-10T03:43:37.610 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: mon.a@-1(???) e1 preinit fsid 414ba1e0-de90-11ee-95c7-87774f69a715 2024-03-10T03:43:37.610 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: mon.a@-1(???).mds e1 new map 2024-03-10T03:43:37.610 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: mon.a@-1(???).mds e1 print_map 2024-03-10T03:43:37.610 INFO:journalctl@ceph.mon.a.smithi006.stdout: e1 2024-03-10T03:43:37.610 INFO:journalctl@ceph.mon.a.smithi006.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-10T03:43:37.610 INFO:journalctl@ceph.mon.a.smithi006.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-10T03:43:37.610 INFO:journalctl@ceph.mon.a.smithi006.stdout: legacy client fscid: -1 2024-03-10T03:43:37.610 INFO:journalctl@ceph.mon.a.smithi006.stdout: 2024-03-10T03:43:37.610 INFO:journalctl@ceph.mon.a.smithi006.stdout: No filesystems configured 2024-03-10T03:43:37.610 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-10T03:43:37.610 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-10T03:43:37.610 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-10T03:43:37.610 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-10T03:43:37.610 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-10T03:43:37.610 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-10T03:43:37.611 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: monmap e1: 1 mons at {a=[v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-10T03:43:37.611 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: fsmap 2024-03-10T03:43:37.611 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: osdmap e1: 0 total, 0 up, 0 in 2024-03-10T03:43:37.611 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:37 smithi006 ceph-mon[25256]: mgrmap e1: no daemons active 2024-03-10T03:43:38.170 INFO:teuthology.orchestra.run.smithi006.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-10T03:43:38.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:38 smithi006 bash[25237]: 95f492a8209d21bc1066387178a63d18d00bc022f85e2ab0cca33b5359771c1d 2024-03-10T03:43:38.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:38 smithi006 systemd[1]: Started Ceph mon.a for 414ba1e0-de90-11ee-95c7-87774f69a715. 2024-03-10T03:43:42.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:42 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/3713240764' entity='client.admin' 2024-03-10T03:43:43.879 INFO:teuthology.orchestra.run.smithi006.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-10T03:43:43.901 INFO:teuthology.orchestra.run.smithi006.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-10T03:43:43.901 INFO:teuthology.orchestra.run.smithi006.stdout:Creating mgr... 2024-03-10T03:43:43.902 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-10T03:43:43.902 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-10T03:43:44.161 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mgr.y 2024-03-10T03:43:44.162 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Failed to reset failed state of unit ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mgr.y.service: Unit ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mgr.y.service not loaded. 2024-03-10T03:43:44.315 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-414ba1e0-de90-11ee-95c7-87774f69a715.target.wants/ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-414ba1e0-de90-11ee-95c7-87774f69a715@.service. 2024-03-10T03:43:44.587 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:44 smithi006 podman[25472]: 2024-03-10 03:43:44.449944496 +0000 UTC m=+0.017552796 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 2024-03-10T03:43:44.794 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-10T03:43:44.794 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout disabled 2024-03-10T03:43:44.801 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-10T03:43:44.801 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout inactive 2024-03-10T03:43:44.801 INFO:teuthology.orchestra.run.smithi006.stdout:firewalld.service is not enabled 2024-03-10T03:43:44.801 INFO:teuthology.orchestra.run.smithi006.stdout:Not possible to enable service . firewalld.service is not available 2024-03-10T03:43:44.810 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-10T03:43:44.810 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout disabled 2024-03-10T03:43:44.817 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-10T03:43:44.818 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout inactive 2024-03-10T03:43:44.818 INFO:teuthology.orchestra.run.smithi006.stdout:firewalld.service is not enabled 2024-03-10T03:43:44.818 INFO:teuthology.orchestra.run.smithi006.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-10T03:43:44.818 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for mgr to start... 2024-03-10T03:43:44.818 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for mgr... 2024-03-10T03:43:44.939 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:44 smithi006 podman[25472]: 2024-03-10 03:43:44.713435917 +0000 UTC m=+0.281044204 container init d42af8aedd6c72411f53bc60c32bd9f1833455fbde23773636b5c3403be7ef30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-10T03:43:44.940 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:44 smithi006 podman[25472]: 2024-03-10 03:43:44.715931925 +0000 UTC m=+0.283540213 container start d42af8aedd6c72411f53bc60c32bd9f1833455fbde23773636b5c3403be7ef30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18) 2024-03-10T03:43:44.940 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:44 smithi006 bash[25472]: d42af8aedd6c72411f53bc60c32bd9f1833455fbde23773636b5c3403be7ef30 2024-03-10T03:43:44.940 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:44 smithi006 systemd[1]: Started Ceph mgr.y for 414ba1e0-de90-11ee-95c7-87774f69a715. 2024-03-10T03:43:45.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:45 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/3666355159' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-10T03:43:45.587 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:45 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:45.285+0000 7fd512c37200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-10T03:43:45.587 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:45 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:45.383+0000 7fd512c37200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-10T03:43:45.908 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-10T03:43:45.908 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-03-10T03:43:45.908 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsid": "414ba1e0-de90-11ee-95c7-87774f69a715", 2024-03-10T03:43:45.909 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "health": { 2024-03-10T03:43:45.909 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-10T03:43:45.909 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-10T03:43:45.909 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-10T03:43:45.909 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:45.909 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-10T03:43:45.909 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-10T03:43:45.909 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 0 2024-03-10T03:43:45.909 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-03-10T03:43:45.909 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-10T03:43:45.909 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "a" 2024-03-10T03:43:45.909 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-03-10T03:43:45.909 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-10T03:43:45.909 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-10T03:43:45.909 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T03:43:45.910 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-10T03:43:45.910 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-10T03:43:45.910 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:45.910 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-10T03:43:45.910 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T03:43:45.910 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-10T03:43:45.910 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-10T03:43:45.910 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-10T03:43:45.910 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-10T03:43:45.910 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-10T03:43:45.910 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-10T03:43:45.913 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:45.914 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-10T03:43:45.914 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-10T03:43:45.914 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-10T03:43:45.914 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-10T03:43:45.914 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-10T03:43:45.914 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-10T03:43:45.914 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-10T03:43:45.914 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-10T03:43:45.914 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-10T03:43:45.914 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:45.914 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-10T03:43:45.915 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T03:43:45.915 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-10T03:43:45.915 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-10T03:43:45.915 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:45.915 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-10T03:43:45.915 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-10T03:43:45.915 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-10T03:43:45.915 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-10T03:43:45.915 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-10T03:43:45.915 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-10T03:43:45.915 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "restful" 2024-03-10T03:43:45.915 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-03-10T03:43:45.915 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-10T03:43:45.916 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:45.916 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-10T03:43:45.916 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T03:43:45.916 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modified": "2024-03-10T03:43:29.585525+0000", 2024-03-10T03:43:45.916 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-10T03:43:45.916 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:45.916 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-10T03:43:45.916 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-03-10T03:43:45.916 INFO:teuthology.orchestra.run.smithi006.stdout:mgr not available, waiting (1/15)... 2024-03-10T03:43:46.336 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:46.079+0000 7fd512c37200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-10T03:43:46.336 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:46.173+0000 7fd512c37200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-10T03:43:46.337 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:46.314+0000 7fd512c37200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-10T03:43:46.763 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:46.412+0000 7fd512c37200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-10T03:43:46.763 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:46.496+0000 7fd512c37200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-10T03:43:46.763 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:46.762+0000 7fd512c37200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-10T03:43:47.058 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:46.873+0000 7fd512c37200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-10T03:43:47.058 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:46.961+0000 7fd512c37200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-10T03:43:47.058 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:47 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:47.057+0000 7fd512c37200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-10T03:43:47.336 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:47 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:47.234+0000 7fd512c37200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-10T03:43:48.336 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:48 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:48.239+0000 7fd512c37200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-10T03:43:48.717 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:48 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/1219474297' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-10T03:43:48.717 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:48 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:48.377+0000 7fd512c37200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-10T03:43:49.009 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:48 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:48.715+0000 7fd512c37200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-10T03:43:49.010 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:48 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:48.807+0000 7fd512c37200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-10T03:43:49.010 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:49 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:49.008+0000 7fd512c37200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-10T03:43:49.336 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:49 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:49.197+0000 7fd512c37200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-10T03:43:49.750 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:49 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:49.487+0000 7fd512c37200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-10T03:43:49.750 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:49 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:49.748+0000 7fd512c37200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-10T03:43:49.905 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-10T03:43:49.905 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-03-10T03:43:49.905 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsid": "414ba1e0-de90-11ee-95c7-87774f69a715", 2024-03-10T03:43:49.905 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "health": { 2024-03-10T03:43:49.905 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-10T03:43:49.905 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-10T03:43:49.905 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-10T03:43:49.905 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:49.905 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-10T03:43:49.905 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-10T03:43:49.906 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 0 2024-03-10T03:43:49.906 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-03-10T03:43:49.906 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-10T03:43:49.906 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "a" 2024-03-10T03:43:49.906 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-03-10T03:43:49.906 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-03-10T03:43:49.906 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-10T03:43:49.906 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T03:43:49.906 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-10T03:43:49.906 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-10T03:43:49.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:49.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-10T03:43:49.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T03:43:49.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-10T03:43:49.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-10T03:43:49.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-10T03:43:49.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-10T03:43:49.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-10T03:43:49.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-10T03:43:49.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:49.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-10T03:43:49.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-10T03:43:49.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-10T03:43:49.910 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-10T03:43:49.911 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-10T03:43:49.911 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-10T03:43:49.911 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-10T03:43:49.911 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-10T03:43:49.911 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-10T03:43:49.911 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:49.911 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-10T03:43:49.911 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T03:43:49.911 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-10T03:43:49.911 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-10T03:43:49.911 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:49.911 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-10T03:43:49.911 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-10T03:43:49.911 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-10T03:43:49.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-10T03:43:49.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-10T03:43:49.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-10T03:43:49.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "restful" 2024-03-10T03:43:49.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-03-10T03:43:49.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-10T03:43:49.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:49.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-10T03:43:49.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T03:43:49.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modified": "2024-03-10T03:43:29.585525+0000", 2024-03-10T03:43:49.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-10T03:43:49.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:49.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-10T03:43:49.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-03-10T03:43:49.912 INFO:teuthology.orchestra.run.smithi006.stdout:mgr not available, waiting (2/15)... 2024-03-10T03:43:50.086 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:49 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:49.993+0000 7fd512c37200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-10T03:43:51.086 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:50 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:50.814+0000 7fd512c37200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-10T03:43:51.711 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:51 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:51.391+0000 7fd512c37200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-10T03:43:51.711 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:51 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:51.491+0000 7fd512c37200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-10T03:43:51.711 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:51 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:51.709+0000 7fd512c37200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-10T03:43:52.028 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:51 smithi006 ceph-mon[25256]: Activating manager daemon y 2024-03-10T03:43:52.028 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:51 smithi006 ceph-mon[25256]: mgrmap e2: y(active, starting, since 0.00308038s) 2024-03-10T03:43:52.028 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:51 smithi006 ceph-mon[25256]: from='mgr.14100 172.21.15.6:0/2164514' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-10T03:43:52.028 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:51 smithi006 ceph-mon[25256]: from='mgr.14100 172.21.15.6:0/2164514' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-10T03:43:52.029 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:51 smithi006 ceph-mon[25256]: from='mgr.14100 172.21.15.6:0/2164514' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-10T03:43:52.029 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:51 smithi006 ceph-mon[25256]: from='mgr.14100 172.21.15.6:0/2164514' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T03:43:52.029 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:51 smithi006 ceph-mon[25256]: from='mgr.14100 172.21.15.6:0/2164514' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-10T03:43:52.029 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:51 smithi006 ceph-mon[25256]: Manager daemon y is now available 2024-03-10T03:43:52.029 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:51 smithi006 ceph-mon[25256]: from='mgr.14100 172.21.15.6:0/2164514' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-10T03:43:52.029 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:51 smithi006 ceph-mon[25256]: from='mgr.14100 172.21.15.6:0/2164514' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-10T03:43:52.029 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:51 smithi006 ceph-mon[25256]: from='mgr.14100 172.21.15.6:0/2164514' entity='mgr.y' 2024-03-10T03:43:52.029 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:51 smithi006 ceph-mon[25256]: from='mgr.14100 172.21.15.6:0/2164514' entity='mgr.y' 2024-03-10T03:43:52.030 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:51 smithi006 ceph-mon[25256]: from='mgr.14100 172.21.15.6:0/2164514' entity='mgr.y' 2024-03-10T03:43:52.935 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-10T03:43:52.935 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-03-10T03:43:52.935 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsid": "414ba1e0-de90-11ee-95c7-87774f69a715", 2024-03-10T03:43:52.935 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "health": { 2024-03-10T03:43:52.935 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-10T03:43:52.935 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-10T03:43:52.935 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-10T03:43:52.935 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:52.935 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-10T03:43:52.936 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-10T03:43:52.936 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 0 2024-03-10T03:43:52.936 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-03-10T03:43:52.936 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-10T03:43:52.936 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "a" 2024-03-10T03:43:52.936 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-03-10T03:43:52.936 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-03-10T03:43:52.936 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-10T03:43:52.936 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T03:43:52.936 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-10T03:43:52.936 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-10T03:43:52.936 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:52.936 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-10T03:43:52.937 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T03:43:52.937 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-10T03:43:52.937 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-10T03:43:52.937 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-10T03:43:52.937 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-10T03:43:52.937 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-10T03:43:52.937 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-10T03:43:52.941 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:52.941 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-10T03:43:52.941 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-10T03:43:52.941 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-10T03:43:52.941 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-10T03:43:52.941 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-10T03:43:52.941 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-10T03:43:52.941 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-10T03:43:52.941 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-10T03:43:52.941 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-10T03:43:52.941 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:52.941 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-10T03:43:52.942 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T03:43:52.942 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-10T03:43:52.942 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-10T03:43:52.942 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:52.942 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-10T03:43:52.942 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-10T03:43:52.942 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-10T03:43:52.942 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-10T03:43:52.942 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-10T03:43:52.942 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-10T03:43:52.942 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "restful" 2024-03-10T03:43:52.942 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-03-10T03:43:52.942 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-10T03:43:52.943 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:52.943 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-10T03:43:52.943 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T03:43:52.943 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modified": "2024-03-10T03:43:29.585525+0000", 2024-03-10T03:43:52.943 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-10T03:43:52.943 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:52.943 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-10T03:43:52.943 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-03-10T03:43:52.943 INFO:teuthology.orchestra.run.smithi006.stdout:mgr not available, waiting (3/15)... 2024-03-10T03:43:53.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:52 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/2970438711' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-10T03:43:53.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:52 smithi006 ceph-mon[25256]: mgrmap e3: y(active, since 1.00847s) 2024-03-10T03:43:55.061 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:54 smithi006 ceph-mon[25256]: mgrmap e4: y(active, since 2s) 2024-03-10T03:43:56.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:55 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/677237960' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-10T03:43:56.315 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-10T03:43:56.316 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-03-10T03:43:56.316 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsid": "414ba1e0-de90-11ee-95c7-87774f69a715", 2024-03-10T03:43:56.316 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "health": { 2024-03-10T03:43:56.316 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-10T03:43:56.316 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-10T03:43:56.316 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-10T03:43:56.316 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:56.316 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-10T03:43:56.316 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-10T03:43:56.316 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 0 2024-03-10T03:43:56.316 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-03-10T03:43:56.316 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-10T03:43:56.316 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "a" 2024-03-10T03:43:56.316 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-03-10T03:43:56.317 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_age": 18, 2024-03-10T03:43:56.317 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-10T03:43:56.317 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T03:43:56.317 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-10T03:43:56.317 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-10T03:43:56.317 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:56.317 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-10T03:43:56.317 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T03:43:56.317 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-10T03:43:56.317 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-10T03:43:56.317 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-10T03:43:56.317 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-10T03:43:56.321 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-10T03:43:56.321 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-10T03:43:56.321 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:56.321 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-10T03:43:56.321 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-10T03:43:56.321 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-10T03:43:56.321 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-10T03:43:56.321 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-10T03:43:56.321 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-10T03:43:56.321 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-10T03:43:56.321 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-10T03:43:56.322 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-10T03:43:56.322 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:56.322 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-10T03:43:56.322 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T03:43:56.322 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-10T03:43:56.322 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-10T03:43:56.322 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:56.322 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-10T03:43:56.322 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-10T03:43:56.322 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-10T03:43:56.322 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-10T03:43:56.322 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-10T03:43:56.322 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-10T03:43:56.322 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "restful" 2024-03-10T03:43:56.323 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-03-10T03:43:56.323 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-10T03:43:56.323 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:56.323 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-10T03:43:56.323 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-10T03:43:56.323 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modified": "2024-03-10T03:43:29.585525+0000", 2024-03-10T03:43:56.323 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-10T03:43:56.323 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-10T03:43:56.323 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-10T03:43:56.323 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-03-10T03:43:56.323 INFO:teuthology.orchestra.run.smithi006.stdout:mgr is available 2024-03-10T03:43:57.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:56 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/3419861558' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-10T03:43:57.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:56 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/3419861558' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-10T03:43:57.545 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-10T03:43:57.545 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout [global] 2024-03-10T03:43:57.545 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout fsid = 414ba1e0-de90-11ee-95c7-87774f69a715 2024-03-10T03:43:57.546 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-10T03:43:57.546 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-10T03:43:57.546 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-10T03:43:57.546 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-10T03:43:57.546 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-10T03:43:57.546 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-10T03:43:57.546 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-10T03:43:57.546 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-10T03:43:57.546 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout [osd] 2024-03-10T03:43:57.546 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-10T03:43:57.546 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-10T03:43:57.546 INFO:teuthology.orchestra.run.smithi006.stdout:Enabling cephadm module... 2024-03-10T03:43:58.580 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:58 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/1253227248' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-10T03:43:58.580 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:58 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: ignoring --setuser ceph since I am not root 2024-03-10T03:43:58.580 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:58 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: ignoring --setgroup ceph since I am not root 2024-03-10T03:43:59.086 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:58 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:58.833+0000 7f078b86a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-10T03:43:59.086 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:58 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:58.935+0000 7f078b86a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-10T03:43:59.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:59 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/1253227248' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-10T03:43:59.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:43:59 smithi006 ceph-mon[25256]: mgrmap e5: y(active, since 6s) 2024-03-10T03:43:59.847 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:59 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:59.598+0000 7f078b86a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-10T03:43:59.847 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:59 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:59.700+0000 7f078b86a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-10T03:43:59.847 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:59 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:59.846+0000 7f078b86a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-10T03:44:00.128 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:43:59 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:43:59.944+0000 7f078b86a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-10T03:44:00.129 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:00 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:00.029+0000 7f078b86a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-10T03:44:00.132 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-03-10T03:44:00.132 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-10T03:44:00.132 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-10T03:44:00.132 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-10T03:44:00.132 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-10T03:44:00.132 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-03-10T03:44:00.132 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for the mgr to restart... 2024-03-10T03:44:00.132 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for mgr epoch 5... 2024-03-10T03:44:00.420 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:00 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/941977640' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-10T03:44:00.420 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:00 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:00.305+0000 7f078b86a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-10T03:44:00.420 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:00 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:00.418+0000 7f078b86a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-10T03:44:00.785 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:00 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:00.509+0000 7f078b86a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-10T03:44:00.785 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:00 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:00.607+0000 7f078b86a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-10T03:44:00.785 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:00 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:00.784+0000 7f078b86a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-10T03:44:02.087 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:01 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:01.758+0000 7f078b86a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-10T03:44:02.087 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:01 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:01.891+0000 7f078b86a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-10T03:44:02.517 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:02 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:02.226+0000 7f078b86a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-10T03:44:02.517 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:02 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:02.318+0000 7f078b86a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-10T03:44:02.517 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:02 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:02.516+0000 7f078b86a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-10T03:44:02.836 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:02 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:02.703+0000 7f078b86a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-10T03:44:03.255 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:02 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:02.992+0000 7f078b86a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-10T03:44:03.255 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:03 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:03.254+0000 7f078b86a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-10T03:44:03.587 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:03 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:03.500+0000 7f078b86a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-10T03:44:04.836 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:04 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:04.428+0000 7f078b86a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-10T03:44:05.329 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:05.009+0000 7f078b86a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-10T03:44:05.329 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:05.109+0000 7f078b86a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-10T03:44:05.329 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:05.327+0000 7f078b86a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-10T03:44:05.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:05 smithi006 ceph-mon[25256]: Active manager daemon y restarted 2024-03-10T03:44:05.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:05 smithi006 ceph-mon[25256]: Activating manager daemon y 2024-03-10T03:44:05.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:05 smithi006 ceph-mon[25256]: osdmap e2: 0 total, 0 up, 0 in 2024-03-10T03:44:05.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:05 smithi006 ceph-mon[25256]: mgrmap e6: y(active, starting, since 0.0036608s) 2024-03-10T03:44:05.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:05 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T03:44:05.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:05 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-10T03:44:05.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:05 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-10T03:44:05.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:05 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-10T03:44:05.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:05 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-10T03:44:05.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:05 smithi006 ceph-mon[25256]: Manager daemon y is now available 2024-03-10T03:44:05.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:05 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' 2024-03-10T03:44:05.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:05 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' 2024-03-10T03:44:05.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:05 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' 2024-03-10T03:44:05.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:05 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:44:05.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:05 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:44:06.703 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:06 smithi006 ceph-mon[25256]: Found migration_current of "None". Setting to last migration. 2024-03-10T03:44:06.703 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:06 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:44:06.704 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:06 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-10T03:44:06.704 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:06 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-10T03:44:06.704 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:06 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' 2024-03-10T03:44:06.704 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:06 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' 2024-03-10T03:44:06.704 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:06 smithi006 ceph-mon[25256]: [10/Mar/2024:03:44:05] ENGINE Bus STARTING 2024-03-10T03:44:06.704 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:06 smithi006 ceph-mon[25256]: [10/Mar/2024:03:44:06] ENGINE Serving on https://172.21.15.6:7150 2024-03-10T03:44:06.704 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:06 smithi006 ceph-mon[25256]: [10/Mar/2024:03:44:06] ENGINE Serving on http://172.21.15.6:8765 2024-03-10T03:44:06.704 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:06 smithi006 ceph-mon[25256]: [10/Mar/2024:03:44:06] ENGINE Bus STARTED 2024-03-10T03:44:06.704 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:06 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:44:06.704 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:06 smithi006 ceph-mon[25256]: mgrmap e7: y(active, since 1.00548s) 2024-03-10T03:44:06.924 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-03-10T03:44:06.925 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-10T03:44:06.925 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-10T03:44:06.925 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-03-10T03:44:06.925 INFO:teuthology.orchestra.run.smithi006.stdout:mgr epoch 5 is available 2024-03-10T03:44:06.925 INFO:teuthology.orchestra.run.smithi006.stdout:Setting orchestrator backend to cephadm... 2024-03-10T03:44:07.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:07 smithi006 ceph-mon[25256]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-10T03:44:07.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:07 smithi006 ceph-mon[25256]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-10T03:44:08.746 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:08 smithi006 ceph-mon[25256]: mgrmap e8: y(active, since 2s) 2024-03-10T03:44:08.746 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:08 smithi006 ceph-mon[25256]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:44:08.746 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:08 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' 2024-03-10T03:44:08.746 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:08 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:44:09.468 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-10T03:44:09.469 INFO:teuthology.orchestra.run.smithi006.stdout:Generating ssh key... 2024-03-10T03:44:09.657 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:09 smithi006 ceph-mon[25256]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:44:10.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:10 smithi006 ceph-mon[25256]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:44:10.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:10 smithi006 ceph-mon[25256]: Generating ssh key... 2024-03-10T03:44:11.288 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:10 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: Generating public/private rsa key pair. 2024-03-10T03:44:11.288 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:10 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: Your identification has been saved in /tmp/tmpveuf9847/key. 2024-03-10T03:44:11.288 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:10 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: Your public key has been saved in /tmp/tmpveuf9847/key.pub. 2024-03-10T03:44:11.288 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:10 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: The key fingerprint is: 2024-03-10T03:44:11.288 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:10 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: SHA256:+Ujx03sZ6oC9Ga1tp4Qz8LVWMBj7K7vdpsSAhlB4HN8 ceph-414ba1e0-de90-11ee-95c7-87774f69a715 2024-03-10T03:44:11.288 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:10 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: The key's randomart image is: 2024-03-10T03:44:11.288 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:10 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: +---[RSA 3072]----+ 2024-03-10T03:44:11.288 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:10 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: | oo. . | 2024-03-10T03:44:11.289 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:10 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: | ..o. . + | 2024-03-10T03:44:11.289 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:10 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: | .. o E o | 2024-03-10T03:44:11.289 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:10 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: | . . = o o | 2024-03-10T03:44:11.289 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:10 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: | . S + + o | 2024-03-10T03:44:11.289 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:10 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: | o B B * o | 2024-03-10T03:44:11.289 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:10 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: | o @ @ o | 2024-03-10T03:44:11.289 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:10 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: | ^.oo | 2024-03-10T03:44:11.289 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:10 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: | *o*=. | 2024-03-10T03:44:11.289 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:10 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: +----[SHA256]-----+ 2024-03-10T03:44:12.198 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:11 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' 2024-03-10T03:44:12.198 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:11 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' 2024-03-10T03:44:12.671 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDDnJgMps10KLezdFlMPQmYOuam+59SRJd78tbIY7ayn4QtNzLUkme/DkbRKH6jEK2Ju4vuau+jVWVijAsGdZYPrd3kGNDdT2gMyaGWjINWySPRln9ILK0T7C8l6flDEhYzz7HOxWWplBlPvoHe9B34793tT5BfxeLjdf0JyLSejv57pCUkvAVkpOudpIjw2n4KmLjXs0ic5EZB8L0MSCiwC2snUtSDdWKP6QqSrlwQSBIFUkZ31ZVRT3eVlA+HafaxCE8o0rzdPlYKDwFiGcIBKRjNNu37eLAnMw1jy3LcI0FwN5mK8nWGuRL2WUAqbOyD09AcIiPAe3JEFLRTtnRrq+fT590b5506QbuxTiYtHerSbNUJUTLJp+JENtkk/G/rfImcGMOEx6iWirnDagOFvczCeyGu9zLAKvtVy/eo84t3Jif0X0j8ZoU0YcEQjlCINQBD+bdhRatwztytTqkSVpbZQN1xM/ebOI33Z4LmdI+ht8yf0SOfCQhCZaH0B0= ceph-414ba1e0-de90-11ee-95c7-87774f69a715 2024-03-10T03:44:12.672 INFO:teuthology.orchestra.run.smithi006.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-10T03:44:12.672 INFO:teuthology.orchestra.run.smithi006.stdout:Adding key to root@localhost authorized_keys... 2024-03-10T03:44:12.672 INFO:teuthology.orchestra.run.smithi006.stdout:Adding host smithi006... 2024-03-10T03:44:13.311 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:12 smithi006 ceph-mon[25256]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:44:14.300 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:13 smithi006 ceph-mon[25256]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi006", "addr": "172.21.15.6", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:44:15.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:14 smithi006 ceph-mon[25256]: Deploying cephadm binary to smithi006 2024-03-10T03:44:18.237 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout Added host 'smithi006' with addr '172.21.15.6' 2024-03-10T03:44:18.238 INFO:teuthology.orchestra.run.smithi006.stdout:Deploying unmanaged mon service... 2024-03-10T03:44:18.807 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:18 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' 2024-03-10T03:44:18.807 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:18 smithi006 ceph-mon[25256]: Added host smithi006 2024-03-10T03:44:18.807 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:18 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:44:19.724 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-10T03:44:19.724 INFO:teuthology.orchestra.run.smithi006.stdout:Deploying unmanaged mgr service... 2024-03-10T03:44:20.508 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:20 smithi006 ceph-mon[25256]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:44:20.508 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:20 smithi006 ceph-mon[25256]: Saving service mon spec with placement count:5 2024-03-10T03:44:20.508 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:20 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' 2024-03-10T03:44:21.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:21 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' 2024-03-10T03:44:21.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:21 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' 2024-03-10T03:44:21.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:21 smithi006 ceph-mon[25256]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:44:21.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:21 smithi006 ceph-mon[25256]: Saving service mgr spec with placement count:2 2024-03-10T03:44:21.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:21 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' 2024-03-10T03:44:21.841 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-10T03:44:23.734 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:23 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/1942422661' entity='client.admin' 2024-03-10T03:44:24.406 INFO:teuthology.orchestra.run.smithi006.stdout:Enabling the dashboard module... 2024-03-10T03:44:25.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:24 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/2109969788' entity='client.admin' 2024-03-10T03:44:25.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:24 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' 2024-03-10T03:44:25.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:24 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' 2024-03-10T03:44:25.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:24 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi006", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:44:25.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:24 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' 2024-03-10T03:44:25.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:24 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi006", "caps": []}]: dispatch 2024-03-10T03:44:25.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:24 smithi006 ceph-mon[25256]: from='mgr.14120 172.21.15.6:0/2248963299' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi006", "caps": []}]': finished 2024-03-10T03:44:26.194 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:25 smithi006 ceph-mon[25256]: Deploying daemon agent.smithi006 on smithi006 2024-03-10T03:44:26.194 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:25 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/2306813202' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-10T03:44:26.194 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:25 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: ignoring --setuser ceph since I am not root 2024-03-10T03:44:26.194 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:25 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: ignoring --setgroup ceph since I am not root 2024-03-10T03:44:26.762 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:26 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:26.453+0000 7fc9cfdea200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-10T03:44:26.762 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:26 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:26.549+0000 7fc9cfdea200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-10T03:44:27.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:26 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/2306813202' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-10T03:44:27.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:26 smithi006 ceph-mon[25256]: mgrmap e9: y(active, since 20s) 2024-03-10T03:44:27.541 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:27 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:27.221+0000 7fc9cfdea200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-10T03:44:27.541 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:27 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:27.320+0000 7fc9cfdea200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-10T03:44:27.541 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:27 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:27.466+0000 7fc9cfdea200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-10T03:44:27.816 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:27 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:27.564+0000 7fc9cfdea200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-10T03:44:27.817 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:27 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:27.649+0000 7fc9cfdea200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-10T03:44:27.819 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-03-10T03:44:27.819 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-10T03:44:27.820 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-10T03:44:27.820 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-10T03:44:27.820 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-10T03:44:27.820 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-03-10T03:44:27.820 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for the mgr to restart... 2024-03-10T03:44:27.820 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for mgr epoch 9... 2024-03-10T03:44:28.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:27 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/1486888264' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-10T03:44:28.087 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:27 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:27.921+0000 7fc9cfdea200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-10T03:44:28.087 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:28 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:28.036+0000 7fc9cfdea200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-10T03:44:28.407 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:28 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:28.126+0000 7fc9cfdea200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-10T03:44:28.407 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:28 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:28.224+0000 7fc9cfdea200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-10T03:44:28.407 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:28 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:28.405+0000 7fc9cfdea200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-10T03:44:29.836 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:29 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:29.390+0000 7fc9cfdea200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-10T03:44:29.836 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:29 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:29.524+0000 7fc9cfdea200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-10T03:44:30.157 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:29 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:29.862+0000 7fc9cfdea200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-10T03:44:30.157 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:29 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:29.954+0000 7fc9cfdea200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-10T03:44:30.157 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:30 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:30.155+0000 7fc9cfdea200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-10T03:44:30.586 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:30 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:30.344+0000 7fc9cfdea200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-10T03:44:30.899 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:30 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:30.636+0000 7fc9cfdea200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-10T03:44:31.336 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:30 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:30.898+0000 7fc9cfdea200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-10T03:44:31.336 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:31 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:31.140+0000 7fc9cfdea200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-10T03:44:32.336 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:31 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:31.965+0000 7fc9cfdea200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-10T03:44:32.836 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:32 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:32.547+0000 7fc9cfdea200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-10T03:44:32.836 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:32 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:32.650+0000 7fc9cfdea200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-10T03:44:33.336 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:44:32 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:44:32.874+0000 7fc9cfdea200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-10T03:44:33.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:32 smithi006 ceph-mon[25256]: Active manager daemon y restarted 2024-03-10T03:44:33.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:32 smithi006 ceph-mon[25256]: Activating manager daemon y 2024-03-10T03:44:33.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:32 smithi006 ceph-mon[25256]: osdmap e3: 0 total, 0 up, 0 in 2024-03-10T03:44:33.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:32 smithi006 ceph-mon[25256]: mgrmap e10: y(active, starting, since 0.00333758s) 2024-03-10T03:44:33.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:32 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T03:44:33.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:32 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-10T03:44:33.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:32 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-10T03:44:33.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:32 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-10T03:44:33.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:32 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-10T03:44:33.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:32 smithi006 ceph-mon[25256]: Manager daemon y is now available 2024-03-10T03:44:33.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:32 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:33.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:32 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:44:33.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:32 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:44:34.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:33 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-10T03:44:34.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:33 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-10T03:44:34.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:33 smithi006 ceph-mon[25256]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-10T03:44:34.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:33 smithi006 ceph-mon[25256]: mgrmap e11: y(active, since 1.00601s) 2024-03-10T03:44:34.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:33 smithi006 ceph-mon[25256]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-10T03:44:34.720 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-03-10T03:44:34.721 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-10T03:44:34.721 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-10T03:44:34.721 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-03-10T03:44:34.721 INFO:teuthology.orchestra.run.smithi006.stdout:mgr epoch 9 is available 2024-03-10T03:44:34.721 INFO:teuthology.orchestra.run.smithi006.stdout:Generating a dashboard self-signed certificate... 2024-03-10T03:44:36.034 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-10T03:44:36.034 INFO:teuthology.orchestra.run.smithi006.stdout:Creating initial admin user... 2024-03-10T03:44:36.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:35 smithi006 ceph-mon[25256]: [10/Mar/2024:03:44:34] ENGINE Bus STARTING 2024-03-10T03:44:36.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:35 smithi006 ceph-mon[25256]: [10/Mar/2024:03:44:34] ENGINE Serving on https://172.21.15.6:7150 2024-03-10T03:44:36.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:35 smithi006 ceph-mon[25256]: [10/Mar/2024:03:44:34] ENGINE Serving on http://172.21.15.6:8765 2024-03-10T03:44:36.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:35 smithi006 ceph-mon[25256]: [10/Mar/2024:03:44:34] ENGINE Bus STARTED 2024-03-10T03:44:36.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:35 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:36.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:35 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi006", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:44:36.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:35 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:36.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:35 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi006", "caps": []}]: dispatch 2024-03-10T03:44:36.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:35 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:36.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:35 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:37.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:36 smithi006 ceph-mon[25256]: Deploying daemon agent.smithi006 on smithi006 2024-03-10T03:44:37.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:36 smithi006 ceph-mon[25256]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:44:37.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:36 smithi006 ceph-mon[25256]: mgrmap e12: y(active, since 2s) 2024-03-10T03:44:37.523 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$nICNfQtIc3jT5BRZNaYuguX14wam2diyxZinWbu0tyX7N3iDgl3aC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710042276, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-10T03:44:37.524 INFO:teuthology.orchestra.run.smithi006.stdout:Fetching dashboard port number... 2024-03-10T03:44:38.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:37 smithi006 ceph-mon[25256]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:44:38.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:37 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:38.705 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 8443 2024-03-10T03:44:38.715 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-10T03:44:38.715 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout disabled 2024-03-10T03:44:38.722 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-10T03:44:38.722 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout inactive 2024-03-10T03:44:38.722 INFO:teuthology.orchestra.run.smithi006.stdout:firewalld.service is not enabled 2024-03-10T03:44:38.722 INFO:teuthology.orchestra.run.smithi006.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-10T03:44:38.724 INFO:teuthology.orchestra.run.smithi006.stdout:Ceph Dashboard is now available at: 2024-03-10T03:44:38.724 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:44:38.725 INFO:teuthology.orchestra.run.smithi006.stdout: URL: https://smithi006.front.sepia.ceph.com:8443/ 2024-03-10T03:44:38.725 INFO:teuthology.orchestra.run.smithi006.stdout: User: admin 2024-03-10T03:44:38.725 INFO:teuthology.orchestra.run.smithi006.stdout: Password: vzjqqmqouf 2024-03-10T03:44:38.725 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:44:38.725 INFO:teuthology.orchestra.run.smithi006.stdout:Saving cluster configuration to /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/config directory 2024-03-10T03:44:38.725 INFO:teuthology.orchestra.run.smithi006.stdout:Enabling autotune for osd_memory_target 2024-03-10T03:44:39.014 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:38 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/2947316514' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-10T03:44:41.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:40 smithi006 ceph-mon[25256]: mgrmap e13: y(active, since 6s) 2024-03-10T03:44:41.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:40 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/313156503' entity='client.admin' 2024-03-10T03:44:41.325 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-10T03:44:41.325 INFO:teuthology.orchestra.run.smithi006.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-10T03:44:41.325 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:44:41.325 INFO:teuthology.orchestra.run.smithi006.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-10T03:44:41.325 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:44:41.325 INFO:teuthology.orchestra.run.smithi006.stdout:Or, if you are only running a single cluster on this host: 2024-03-10T03:44:41.325 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:44:41.326 INFO:teuthology.orchestra.run.smithi006.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-10T03:44:41.326 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:44:41.326 INFO:teuthology.orchestra.run.smithi006.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-10T03:44:41.326 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:44:41.326 INFO:teuthology.orchestra.run.smithi006.stdout: ceph telemetry on 2024-03-10T03:44:41.326 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:44:41.326 INFO:teuthology.orchestra.run.smithi006.stdout:For more information see: 2024-03-10T03:44:41.326 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:44:41.326 INFO:teuthology.orchestra.run.smithi006.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-10T03:44:41.326 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:44:41.326 INFO:teuthology.orchestra.run.smithi006.stdout:Bootstrap complete. 2024-03-10T03:44:41.367 INFO:tasks.cephadm:Fetching config... 2024-03-10T03:44:41.367 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-10T03:44:41.367 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-10T03:44:41.382 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-10T03:44:41.382 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-10T03:44:41.382 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-10T03:44:41.437 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-10T03:44:41.437 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-10T03:44:41.437 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-10T03:44:41.607 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-10T03:44:41.607 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-10T03:44:41.607 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-10T03:44:41.661 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-10T03:44:41.661 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDDnJgMps10KLezdFlMPQmYOuam+59SRJd78tbIY7ayn4QtNzLUkme/DkbRKH6jEK2Ju4vuau+jVWVijAsGdZYPrd3kGNDdT2gMyaGWjINWySPRln9ILK0T7C8l6flDEhYzz7HOxWWplBlPvoHe9B34793tT5BfxeLjdf0JyLSejv57pCUkvAVkpOudpIjw2n4KmLjXs0ic5EZB8L0MSCiwC2snUtSDdWKP6QqSrlwQSBIFUkZ31ZVRT3eVlA+HafaxCE8o0rzdPlYKDwFiGcIBKRjNNu37eLAnMw1jy3LcI0FwN5mK8nWGuRL2WUAqbOyD09AcIiPAe3JEFLRTtnRrq+fT590b5506QbuxTiYtHerSbNUJUTLJp+JENtkk/G/rfImcGMOEx6iWirnDagOFvczCeyGu9zLAKvtVy/eo84t3Jif0X0j8ZoU0YcEQjlCINQBD+bdhRatwztytTqkSVpbZQN1xM/ebOI33Z4LmdI+ht8yf0SOfCQhCZaH0B0= ceph-414ba1e0-de90-11ee-95c7-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-10T03:44:41.825 INFO:teuthology.orchestra.run.smithi006.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDDnJgMps10KLezdFlMPQmYOuam+59SRJd78tbIY7ayn4QtNzLUkme/DkbRKH6jEK2Ju4vuau+jVWVijAsGdZYPrd3kGNDdT2gMyaGWjINWySPRln9ILK0T7C8l6flDEhYzz7HOxWWplBlPvoHe9B34793tT5BfxeLjdf0JyLSejv57pCUkvAVkpOudpIjw2n4KmLjXs0ic5EZB8L0MSCiwC2snUtSDdWKP6QqSrlwQSBIFUkZ31ZVRT3eVlA+HafaxCE8o0rzdPlYKDwFiGcIBKRjNNu37eLAnMw1jy3LcI0FwN5mK8nWGuRL2WUAqbOyD09AcIiPAe3JEFLRTtnRrq+fT590b5506QbuxTiYtHerSbNUJUTLJp+JENtkk/G/rfImcGMOEx6iWirnDagOFvczCeyGu9zLAKvtVy/eo84t3Jif0X0j8ZoU0YcEQjlCINQBD+bdhRatwztytTqkSVpbZQN1xM/ebOI33Z4LmdI+ht8yf0SOfCQhCZaH0B0= ceph-414ba1e0-de90-11ee-95c7-87774f69a715 2024-03-10T03:44:41.840 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDDnJgMps10KLezdFlMPQmYOuam+59SRJd78tbIY7ayn4QtNzLUkme/DkbRKH6jEK2Ju4vuau+jVWVijAsGdZYPrd3kGNDdT2gMyaGWjINWySPRln9ILK0T7C8l6flDEhYzz7HOxWWplBlPvoHe9B34793tT5BfxeLjdf0JyLSejv57pCUkvAVkpOudpIjw2n4KmLjXs0ic5EZB8L0MSCiwC2snUtSDdWKP6QqSrlwQSBIFUkZ31ZVRT3eVlA+HafaxCE8o0rzdPlYKDwFiGcIBKRjNNu37eLAnMw1jy3LcI0FwN5mK8nWGuRL2WUAqbOyD09AcIiPAe3JEFLRTtnRrq+fT590b5506QbuxTiYtHerSbNUJUTLJp+JENtkk/G/rfImcGMOEx6iWirnDagOFvczCeyGu9zLAKvtVy/eo84t3Jif0X0j8ZoU0YcEQjlCINQBD+bdhRatwztytTqkSVpbZQN1xM/ebOI33Z4LmdI+ht8yf0SOfCQhCZaH0B0= ceph-414ba1e0-de90-11ee-95c7-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-10T03:44:41.885 INFO:teuthology.orchestra.run.smithi071.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDDnJgMps10KLezdFlMPQmYOuam+59SRJd78tbIY7ayn4QtNzLUkme/DkbRKH6jEK2Ju4vuau+jVWVijAsGdZYPrd3kGNDdT2gMyaGWjINWySPRln9ILK0T7C8l6flDEhYzz7HOxWWplBlPvoHe9B34793tT5BfxeLjdf0JyLSejv57pCUkvAVkpOudpIjw2n4KmLjXs0ic5EZB8L0MSCiwC2snUtSDdWKP6QqSrlwQSBIFUkZ31ZVRT3eVlA+HafaxCE8o0rzdPlYKDwFiGcIBKRjNNu37eLAnMw1jy3LcI0FwN5mK8nWGuRL2WUAqbOyD09AcIiPAe3JEFLRTtnRrq+fT590b5506QbuxTiYtHerSbNUJUTLJp+JENtkk/G/rfImcGMOEx6iWirnDagOFvczCeyGu9zLAKvtVy/eo84t3Jif0X0j8ZoU0YcEQjlCINQBD+bdhRatwztytTqkSVpbZQN1xM/ebOI33Z4LmdI+ht8yf0SOfCQhCZaH0B0= ceph-414ba1e0-de90-11ee-95c7-87774f69a715 2024-03-10T03:44:41.900 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-10T03:44:42.160 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:44:43.510 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:43.511 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:43.511 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:43.511 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:44:43.511 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:43.511 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:43.511 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:43.511 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:43.511 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:43.511 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:44:43.511 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:43.511 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:43.511 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:45.006 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-10T03:44:45.006 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-10T03:44:45.225 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:44 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/2708662197' entity='client.admin' 2024-03-10T03:44:45.225 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:44 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:44:45.226 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:44 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:46.036 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:44:47.530 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:47 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:47.530 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:47 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:49.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:48 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:49.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:48 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:49.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:48 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:49.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:48 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:44:49.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:48 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:44:49.321 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:48 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:44:49.645 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi071 2024-03-10T03:44:49.645 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-10T03:44:49.645 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/etc/ceph/ceph.conf 2024-03-10T03:44:49.664 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-10T03:44:49.664 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-10T03:44:49.720 INFO:tasks.cephadm:Adding host smithi071 to orchestrator... 2024-03-10T03:44:49.721 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph orch host add smithi071 2024-03-10T03:44:49.960 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:44:50.180 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:49 smithi006 ceph-mon[25256]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:44:50.180 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:49 smithi006 ceph-mon[25256]: Updating smithi006:/etc/ceph/ceph.conf 2024-03-10T03:44:50.180 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:49 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:50.180 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:49 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:50.180 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:49 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:51.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:50 smithi006 ceph-mon[25256]: Updating smithi006:/var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T03:44:51.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:50 smithi006 ceph-mon[25256]: Updating smithi006:/etc/ceph/ceph.client.admin.keyring 2024-03-10T03:44:51.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:50 smithi006 ceph-mon[25256]: Updating smithi006:/var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/config/ceph.client.admin.keyring 2024-03-10T03:44:52.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:51 smithi006 ceph-mon[25256]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi071", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:44:53.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:52 smithi006 ceph-mon[25256]: Deploying cephadm binary to smithi071 2024-03-10T03:44:53.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:52 smithi006 ceph-mon[25256]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:44:55.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:54 smithi006 ceph-mon[25256]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:44:56.061 INFO:teuthology.orchestra.run.smithi006.stdout:Added host 'smithi071' with addr '172.21.15.71' 2024-03-10T03:44:56.652 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph orch host ls --format=json 2024-03-10T03:44:56.895 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:44:57.298 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:57 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:57.298 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:57 smithi006 ceph-mon[25256]: Added host smithi071 2024-03-10T03:44:57.298 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:57 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:44:57.298 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:57 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:57.298 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:57 smithi006 ceph-mon[25256]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:44:57.298 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:57 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:57.299 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:57 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:57.299 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:57 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi071", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:44:57.299 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:57 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:44:57.299 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:57 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:44:58.706 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:44:58.706 INFO:teuthology.orchestra.run.smithi006.stdout:[{"addr": "172.21.15.6", "hostname": "smithi006", "labels": [], "status": ""}, {"addr": "172.21.15.71", "hostname": "smithi071", "labels": [], "status": ""}] 2024-03-10T03:44:59.033 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:58 smithi006 ceph-mon[25256]: Updating smithi071:/etc/ceph/ceph.conf 2024-03-10T03:44:59.033 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:58 smithi006 ceph-mon[25256]: Updating smithi071:/var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T03:44:59.033 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:58 smithi006 ceph-mon[25256]: Updating smithi071:/etc/ceph/ceph.client.admin.keyring 2024-03-10T03:44:59.033 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:58 smithi006 ceph-mon[25256]: Updating smithi071:/var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/config/ceph.client.admin.keyring 2024-03-10T03:44:59.033 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:58 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:59.033 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:58 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:44:59.033 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:58 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi071", "caps": []}]: dispatch 2024-03-10T03:44:59.033 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:58 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi071", "caps": []}]': finished 2024-03-10T03:44:59.353 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-10T03:44:59.353 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph osd crush tunables default 2024-03-10T03:44:59.588 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:45:00.003 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:59 smithi006 ceph-mon[25256]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-10T03:45:00.003 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:59 smithi006 ceph-mon[25256]: Deploying daemon agent.smithi071 on smithi071 2024-03-10T03:45:00.003 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:59 smithi006 ceph-mon[25256]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:00.003 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:59 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:00.003 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:59 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:00.003 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:59 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:00.003 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:59 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:00.003 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:59 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:00.003 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:59 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:00.004 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:59 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:00.004 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:59 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:00.004 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:59 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:00.004 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:59 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:00.004 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:44:59 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:01.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:01 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:01.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:01 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:01.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:01 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:01.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:01 smithi006 ceph-mon[25256]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:02.207 INFO:teuthology.orchestra.run.smithi006.stderr:adjusted tunables profile to default 2024-03-10T03:45:02.495 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:02 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/2497286814' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-10T03:45:02.737 INFO:tasks.cephadm:Adding mon.a on smithi006 2024-03-10T03:45:02.737 INFO:tasks.cephadm:Adding mon.c on smithi006 2024-03-10T03:45:02.737 INFO:tasks.cephadm:Adding mon.b on smithi071 2024-03-10T03:45:02.737 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph orch apply mon '3;smithi006:172.21.15.6=a;smithi006:[v2:172.21.15.6:3301,v1:172.21.15.6:6790]=c;smithi071:172.21.15.71=b' 2024-03-10T03:45:03.084 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T03:45:03.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:03 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/2497286814' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-10T03:45:03.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:03 smithi006 ceph-mon[25256]: osdmap e4: 0 total, 0 up, 0 in 2024-03-10T03:45:03.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:03 smithi006 ceph-mon[25256]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:03.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:03 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:03.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:03 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:03.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:03 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:04.519 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:04 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:04.519 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:04 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:04.519 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:04 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:04.770 INFO:teuthology.orchestra.run.smithi071.stdout:Scheduled mon update... 2024-03-10T03:45:05.368 DEBUG:teuthology.orchestra.run.smithi006:mon.c> sudo journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mon.c.service 2024-03-10T03:45:05.371 DEBUG:teuthology.orchestra.run.smithi071:mon.b> sudo journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mon.b.service 2024-03-10T03:45:05.373 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-10T03:45:05.373 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph mon dump -f json 2024-03-10T03:45:06.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:05 smithi006 ceph-mon[25256]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi006:172.21.15.6=a;smithi006:[v2:172.21.15.6:3301,v1:172.21.15.6:6790]=c;smithi071:172.21.15.71=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:45:06.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:05 smithi006 ceph-mon[25256]: Saving service mon spec with placement smithi006:172.21.15.6=a;smithi006:[v2:172.21.15.6:3301,v1:172.21.15.6:6790]=c;smithi071:172.21.15.71=b;count:3 2024-03-10T03:45:06.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:05 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:06.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:05 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:06.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:05 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:06.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:05 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:06.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:05 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:06.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:05 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T03:45:06.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:05 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:06.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:05 smithi006 ceph-mon[25256]: Deploying daemon mon.b on smithi071 2024-03-10T03:45:06.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:05 smithi006 ceph-mon[25256]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:06.598 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T03:45:07.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:06 smithi006 ceph-mon[25256]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:08.976 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:08 smithi071 systemd[1]: Starting Ceph mon.b for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:45:09.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:08 smithi006 ceph-mon[25256]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:09.554 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 podman[27140]: 2024-03-10 03:45:09.383162303 +0000 UTC m=+0.101490961 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada 2024-03-10T03:45:09.554 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 podman[27140]: 2024-03-10 03:45:09.553401032 +0000 UTC m=+0.271729658 container create e6f6d97fafa3d71628a42c4a3e70c0bdbb34c30acc54c464df0282cf620c191d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240222, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-10T03:45:09.779 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-10T03:45:09.779 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":1,"fsid":"414ba1e0-de90-11ee-95c7-87774f69a715","modified":"2024-03-10T03:43:26.668413Z","created":"2024-03-10T03:43:26.668413Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-10T03:45:09.780 INFO:teuthology.orchestra.run.smithi071.stderr:dumped monmap epoch 1 2024-03-10T03:45:09.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 podman[27140]: 2024-03-10 03:45:09.730508514 +0000 UTC m=+0.448837155 container init e6f6d97fafa3d71628a42c4a3e70c0bdbb34c30acc54c464df0282cf620c191d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True) 2024-03-10T03:45:09.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 podman[27140]: 2024-03-10 03:45:09.733733713 +0000 UTC m=+0.452062343 container start e6f6d97fafa3d71628a42c4a3e70c0bdbb34c30acc54c464df0282cf620c191d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-b, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.build-date=20240222, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-10T03:45:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: set uid:gid to 167:167 (ceph:ceph) 2024-03-10T03:45:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: ceph version 19.0.0-1875-g831bc188 (831bc188fa69f143bbba69ac76119e784500e8e3) squid (dev), process ceph-mon, pid 2 2024-03-10T03:45:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: pidfile_write: ignore empty --pid-file 2024-03-10T03:45:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: load: jerasure load: lrc 2024-03-10T03:45:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: RocksDB version: 7.9.2 2024-03-10T03:45:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Git sha 0 2024-03-10T03:45:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Compile date 2024-03-06 01:10:31 2024-03-10T03:45:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: DB SUMMARY 2024-03-10T03:45:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: DB Session ID: 45LSISL5C3Y45E3LYWXE 2024-03-10T03:45:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: CURRENT file: CURRENT 2024-03-10T03:45:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: IDENTITY file: IDENTITY 2024-03-10T03:45:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-10T03:45:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-10T03:45:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-03-10T03:45:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.error_if_exists: 0 2024-03-10T03:45:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.create_if_missing: 0 2024-03-10T03:45:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.paranoid_checks: 1 2024-03-10T03:45:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-10T03:45:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-10T03:45:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-10T03:45:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.env: 0x564414df0b20 2024-03-10T03:45:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.fs: PosixFileSystem 2024-03-10T03:45:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.info_log: 0x564416aa18e0 2024-03-10T03:45:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_file_opening_threads: 16 2024-03-10T03:45:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.statistics: (nil) 2024-03-10T03:45:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.use_fsync: 0 2024-03-10T03:45:09.844 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_log_file_size: 0 2024-03-10T03:45:09.844 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-10T03:45:09.844 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-10T03:45:09.844 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.keep_log_file_num: 1000 2024-03-10T03:45:09.844 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.recycle_log_file_num: 0 2024-03-10T03:45:09.844 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.allow_fallocate: 1 2024-03-10T03:45:09.844 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.allow_mmap_reads: 0 2024-03-10T03:45:09.844 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.allow_mmap_writes: 0 2024-03-10T03:45:09.844 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.use_direct_reads: 0 2024-03-10T03:45:09.844 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-10T03:45:09.844 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.create_missing_column_families: 0 2024-03-10T03:45:09.844 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.db_log_dir: 2024-03-10T03:45:09.844 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.wal_dir: 2024-03-10T03:45:09.844 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-10T03:45:09.845 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-10T03:45:09.845 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-10T03:45:09.845 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-10T03:45:09.845 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-10T03:45:09.845 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-10T03:45:09.845 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.advise_random_on_open: 1 2024-03-10T03:45:09.845 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.db_write_buffer_size: 0 2024-03-10T03:45:09.845 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.write_buffer_manager: 0x564416a75400 2024-03-10T03:45:09.845 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-10T03:45:09.845 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-10T03:45:09.845 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-10T03:45:09.845 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.rate_limiter: (nil) 2024-03-10T03:45:09.845 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-10T03:45:09.845 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.wal_recovery_mode: 2 2024-03-10T03:45:09.846 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.enable_thread_tracking: 0 2024-03-10T03:45:09.846 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.enable_pipelined_write: 0 2024-03-10T03:45:09.846 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.unordered_write: 0 2024-03-10T03:45:09.846 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-10T03:45:09.846 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-10T03:45:09.846 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-10T03:45:09.846 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-10T03:45:09.846 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.row_cache: None 2024-03-10T03:45:09.846 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.wal_filter: None 2024-03-10T03:45:09.847 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-10T03:45:09.848 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.allow_ingest_behind: 0 2024-03-10T03:45:09.848 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.two_write_queues: 0 2024-03-10T03:45:09.848 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.manual_wal_flush: 0 2024-03-10T03:45:09.848 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.wal_compression: 0 2024-03-10T03:45:09.848 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.atomic_flush: 0 2024-03-10T03:45:09.848 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-10T03:45:09.848 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-10T03:45:09.848 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-10T03:45:09.848 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.log_readahead_size: 0 2024-03-10T03:45:09.848 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-10T03:45:09.848 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.best_efforts_recovery: 0 2024-03-10T03:45:09.848 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-10T03:45:09.848 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-10T03:45:09.849 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.allow_data_in_errors: 0 2024-03-10T03:45:09.849 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.db_host_id: __hostname__ 2024-03-10T03:45:09.849 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-10T03:45:09.849 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_background_jobs: 2 2024-03-10T03:45:09.849 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_background_compactions: -1 2024-03-10T03:45:09.849 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_subcompactions: 1 2024-03-10T03:45:09.849 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-10T03:45:09.849 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-10T03:45:09.849 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-10T03:45:09.849 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_total_wal_size: 0 2024-03-10T03:45:09.849 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-10T03:45:09.849 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-10T03:45:09.849 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-10T03:45:09.849 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-10T03:45:09.850 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_open_files: -1 2024-03-10T03:45:09.850 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.bytes_per_sync: 0 2024-03-10T03:45:09.850 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-10T03:45:09.850 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-10T03:45:09.850 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.compaction_readahead_size: 0 2024-03-10T03:45:09.850 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_background_flushes: -1 2024-03-10T03:45:09.850 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Compression algorithms supported: 2024-03-10T03:45:09.850 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-10T03:45:09.853 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: kZSTD supported: 0 2024-03-10T03:45:09.853 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: kXpressCompression supported: 0 2024-03-10T03:45:09.853 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: kLZ4HCCompression supported: 1 2024-03-10T03:45:09.853 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: kZlibCompression supported: 1 2024-03-10T03:45:09.853 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: kSnappyCompression supported: 1 2024-03-10T03:45:09.853 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: kLZ4Compression supported: 1 2024-03-10T03:45:09.853 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: kBZip2Compression supported: 0 2024-03-10T03:45:09.853 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-10T03:45:09.854 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-10T03:45:09.854 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-10T03:45:09.854 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-10T03:45:09.854 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-10T03:45:09.854 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.merge_operator: 2024-03-10T03:45:09.854 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.compaction_filter: None 2024-03-10T03:45:09.854 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.compaction_filter_factory: None 2024-03-10T03:45:09.854 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.sst_partitioner_factory: None 2024-03-10T03:45:09.854 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-10T03:45:09.854 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-10T03:45:09.854 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564416aa19e0) 2024-03-10T03:45:09.854 INFO:journalctl@ceph.mon.b.smithi071.stdout: cache_index_and_filter_blocks: 1 2024-03-10T03:45:09.854 INFO:journalctl@ceph.mon.b.smithi071.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-10T03:45:09.854 INFO:journalctl@ceph.mon.b.smithi071.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-10T03:45:09.855 INFO:journalctl@ceph.mon.b.smithi071.stdout: pin_top_level_index_and_filter: 1 2024-03-10T03:45:09.855 INFO:journalctl@ceph.mon.b.smithi071.stdout: index_type: 0 2024-03-10T03:45:09.855 INFO:journalctl@ceph.mon.b.smithi071.stdout: data_block_index_type: 0 2024-03-10T03:45:09.855 INFO:journalctl@ceph.mon.b.smithi071.stdout: index_shortening: 1 2024-03-10T03:45:09.855 INFO:journalctl@ceph.mon.b.smithi071.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-10T03:45:09.855 INFO:journalctl@ceph.mon.b.smithi071.stdout: checksum: 4 2024-03-10T03:45:09.855 INFO:journalctl@ceph.mon.b.smithi071.stdout: no_block_cache: 0 2024-03-10T03:45:09.855 INFO:journalctl@ceph.mon.b.smithi071.stdout: block_cache: 0x564416aa7090 2024-03-10T03:45:09.855 INFO:journalctl@ceph.mon.b.smithi071.stdout: block_cache_name: BinnedLRUCache 2024-03-10T03:45:09.855 INFO:journalctl@ceph.mon.b.smithi071.stdout: block_cache_options: 2024-03-10T03:45:09.855 INFO:journalctl@ceph.mon.b.smithi071.stdout: capacity : 536870912 2024-03-10T03:45:09.855 INFO:journalctl@ceph.mon.b.smithi071.stdout: num_shard_bits : 4 2024-03-10T03:45:09.855 INFO:journalctl@ceph.mon.b.smithi071.stdout: strict_capacity_limit : 0 2024-03-10T03:45:09.855 INFO:journalctl@ceph.mon.b.smithi071.stdout: high_pri_pool_ratio: 0.000 2024-03-10T03:45:09.855 INFO:journalctl@ceph.mon.b.smithi071.stdout: block_cache_compressed: (nil) 2024-03-10T03:45:09.856 INFO:journalctl@ceph.mon.b.smithi071.stdout: persistent_cache: (nil) 2024-03-10T03:45:09.856 INFO:journalctl@ceph.mon.b.smithi071.stdout: block_size: 4096 2024-03-10T03:45:09.856 INFO:journalctl@ceph.mon.b.smithi071.stdout: block_size_deviation: 10 2024-03-10T03:45:09.856 INFO:journalctl@ceph.mon.b.smithi071.stdout: block_restart_interval: 16 2024-03-10T03:45:09.856 INFO:journalctl@ceph.mon.b.smithi071.stdout: index_block_restart_interval: 1 2024-03-10T03:45:09.856 INFO:journalctl@ceph.mon.b.smithi071.stdout: metadata_block_size: 4096 2024-03-10T03:45:09.856 INFO:journalctl@ceph.mon.b.smithi071.stdout: partition_filters: 0 2024-03-10T03:45:09.856 INFO:journalctl@ceph.mon.b.smithi071.stdout: use_delta_encoding: 1 2024-03-10T03:45:09.856 INFO:journalctl@ceph.mon.b.smithi071.stdout: filter_policy: bloomfilter 2024-03-10T03:45:09.856 INFO:journalctl@ceph.mon.b.smithi071.stdout: whole_key_filtering: 1 2024-03-10T03:45:09.856 INFO:journalctl@ceph.mon.b.smithi071.stdout: verify_compression: 0 2024-03-10T03:45:09.856 INFO:journalctl@ceph.mon.b.smithi071.stdout: read_amp_bytes_per_bit: 0 2024-03-10T03:45:09.856 INFO:journalctl@ceph.mon.b.smithi071.stdout: format_version: 5 2024-03-10T03:45:09.856 INFO:journalctl@ceph.mon.b.smithi071.stdout: enable_index_compression: 1 2024-03-10T03:45:09.856 INFO:journalctl@ceph.mon.b.smithi071.stdout: block_align: 0 2024-03-10T03:45:09.857 INFO:journalctl@ceph.mon.b.smithi071.stdout: max_auto_readahead_size: 262144 2024-03-10T03:45:09.857 INFO:journalctl@ceph.mon.b.smithi071.stdout: prepopulate_block_cache: 0 2024-03-10T03:45:09.857 INFO:journalctl@ceph.mon.b.smithi071.stdout: initial_auto_readahead_size: 8192 2024-03-10T03:45:09.857 INFO:journalctl@ceph.mon.b.smithi071.stdout: num_file_reads_for_auto_readahead: 2 2024-03-10T03:45:09.857 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.write_buffer_size: 33554432 2024-03-10T03:45:09.857 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_write_buffer_number: 2 2024-03-10T03:45:09.857 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.compression: NoCompression 2024-03-10T03:45:09.857 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.bottommost_compression: Disabled 2024-03-10T03:45:09.857 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.prefix_extractor: nullptr 2024-03-10T03:45:09.857 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-10T03:45:09.857 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.num_levels: 7 2024-03-10T03:45:09.857 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-10T03:45:09.857 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-10T03:45:09.857 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-10T03:45:09.857 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-10T03:45:09.858 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-10T03:45:09.858 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-10T03:45:09.858 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-10T03:45:09.858 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-10T03:45:09.858 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-10T03:45:09.858 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-10T03:45:09.858 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-10T03:45:09.858 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-10T03:45:09.858 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-10T03:45:09.858 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.compression_opts.level: 32767 2024-03-10T03:45:09.858 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.compression_opts.strategy: 0 2024-03-10T03:45:09.858 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-10T03:45:09.858 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-10T03:45:09.858 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-10T03:45:09.859 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-10T03:45:09.859 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.compression_opts.enabled: false 2024-03-10T03:45:09.859 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-10T03:45:09.859 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-10T03:45:09.859 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-10T03:45:09.859 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-10T03:45:09.859 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.target_file_size_base: 67108864 2024-03-10T03:45:09.859 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-10T03:45:09.859 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-10T03:45:09.859 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-10T03:45:09.859 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-10T03:45:09.859 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-10T03:45:09.859 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-10T03:45:09.859 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-10T03:45:09.859 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-10T03:45:09.860 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-10T03:45:09.860 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-10T03:45:09.860 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-10T03:45:09.860 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-10T03:45:09.860 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-10T03:45:09.860 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-10T03:45:09.860 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.arena_block_size: 1048576 2024-03-10T03:45:09.860 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-10T03:45:09.860 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-10T03:45:09.860 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.disable_auto_compactions: 0 2024-03-10T03:45:09.860 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-10T03:45:09.860 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-10T03:45:09.860 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-10T03:45:09.860 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-10T03:45:09.860 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-10T03:45:09.861 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-10T03:45:09.861 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-10T03:45:09.861 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-10T03:45:09.861 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-10T03:45:09.861 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-10T03:45:09.861 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.table_properties_collectors: 2024-03-10T03:45:09.861 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.inplace_update_support: 0 2024-03-10T03:45:09.861 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-10T03:45:09.861 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-10T03:45:09.861 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-10T03:45:09.861 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-10T03:45:09.861 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.bloom_locality: 0 2024-03-10T03:45:09.861 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.max_successive_merges: 0 2024-03-10T03:45:09.861 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-10T03:45:09.862 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.paranoid_file_checks: 0 2024-03-10T03:45:09.862 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.force_consistency_checks: 1 2024-03-10T03:45:09.862 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.report_bg_io_stats: 0 2024-03-10T03:45:09.862 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.ttl: 2592000 2024-03-10T03:45:09.862 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-10T03:45:09.862 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-10T03:45:09.862 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-10T03:45:09.862 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.enable_blob_files: false 2024-03-10T03:45:09.862 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.min_blob_size: 0 2024-03-10T03:45:09.862 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.blob_file_size: 268435456 2024-03-10T03:45:09.862 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-10T03:45:09.862 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-10T03:45:09.862 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-10T03:45:09.863 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-10T03:45:09.863 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-10T03:45:09.863 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.blob_file_starting_level: 0 2024-03-10T03:45:09.863 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-10T03:45:09.863 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-10T03:45:09.863 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-10T03:45:09.863 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: cdf3e042-dec1-49d9-9a49-f47fb497eccb 2024-03-10T03:45:09.863 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710042309773999, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-10T03:45:09.863 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-10T03:45:09.863 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710042309774408, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710042309, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "cdf3e042-dec1-49d9-9a49-f47fb497eccb", "db_session_id": "45LSISL5C3Y45E3LYWXE", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-10T03:45:09.863 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710042309774487, "job": 1, "event": "recovery_finished"} 2024-03-10T03:45:09.863 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-10T03:45:09.863 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-10T03:45:09.863 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564416b9a000 2024-03-10T03:45:09.864 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: DB pointer 0x564416b84000 2024-03-10T03:45:09.864 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-10T03:45:09.864 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-10T03:45:09.864 INFO:journalctl@ceph.mon.b.smithi071.stdout: ** DB Stats ** 2024-03-10T03:45:09.864 INFO:journalctl@ceph.mon.b.smithi071.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-10T03:45:09.864 INFO:journalctl@ceph.mon.b.smithi071.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-10T03:45:09.864 INFO:journalctl@ceph.mon.b.smithi071.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-10T03:45:09.864 INFO:journalctl@ceph.mon.b.smithi071.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-10T03:45:09.864 INFO:journalctl@ceph.mon.b.smithi071.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-10T03:45:09.864 INFO:journalctl@ceph.mon.b.smithi071.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-10T03:45:09.864 INFO:journalctl@ceph.mon.b.smithi071.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-10T03:45:09.864 INFO:journalctl@ceph.mon.b.smithi071.stdout: 2024-03-10T03:45:09.864 INFO:journalctl@ceph.mon.b.smithi071.stdout: ** Compaction Stats [default] ** 2024-03-10T03:45:09.864 INFO:journalctl@ceph.mon.b.smithi071.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-10T03:45:09.865 INFO:journalctl@ceph.mon.b.smithi071.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-10T03:45:09.865 INFO:journalctl@ceph.mon.b.smithi071.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-10T03:45:09.865 INFO:journalctl@ceph.mon.b.smithi071.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-10T03:45:09.865 INFO:journalctl@ceph.mon.b.smithi071.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-10T03:45:09.865 INFO:journalctl@ceph.mon.b.smithi071.stdout: 2024-03-10T03:45:09.865 INFO:journalctl@ceph.mon.b.smithi071.stdout: ** Compaction Stats [default] ** 2024-03-10T03:45:09.865 INFO:journalctl@ceph.mon.b.smithi071.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-10T03:45:09.865 INFO:journalctl@ceph.mon.b.smithi071.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-10T03:45:09.865 INFO:journalctl@ceph.mon.b.smithi071.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-10T03:45:09.865 INFO:journalctl@ceph.mon.b.smithi071.stdout: 2024-03-10T03:45:09.865 INFO:journalctl@ceph.mon.b.smithi071.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-10T03:45:09.865 INFO:journalctl@ceph.mon.b.smithi071.stdout: 2024-03-10T03:45:09.865 INFO:journalctl@ceph.mon.b.smithi071.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-10T03:45:09.865 INFO:journalctl@ceph.mon.b.smithi071.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-10T03:45:09.866 INFO:journalctl@ceph.mon.b.smithi071.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-10T03:45:09.866 INFO:journalctl@ceph.mon.b.smithi071.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-10T03:45:09.866 INFO:journalctl@ceph.mon.b.smithi071.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-10T03:45:09.866 INFO:journalctl@ceph.mon.b.smithi071.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-10T03:45:09.866 INFO:journalctl@ceph.mon.b.smithi071.stdout: Cumulative compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-10T03:45:09.866 INFO:journalctl@ceph.mon.b.smithi071.stdout: Interval compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-10T03:45:09.866 INFO:journalctl@ceph.mon.b.smithi071.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-10T03:45:09.866 INFO:journalctl@ceph.mon.b.smithi071.stdout: Block cache BinnedLRUCache@0x564416aa7090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-10T03:45:09.866 INFO:journalctl@ceph.mon.b.smithi071.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-10T03:45:09.866 INFO:journalctl@ceph.mon.b.smithi071.stdout: 2024-03-10T03:45:09.866 INFO:journalctl@ceph.mon.b.smithi071.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-10T03:45:09.866 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-10T03:45:09.866 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: using public_addr v2:172.21.15.71:0/0 -> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-03-10T03:45:09.866 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: starting mon.b rank -1 at public addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] at bind addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 414ba1e0-de90-11ee-95c7-87774f69a715 2024-03-10T03:45:09.867 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: mon.b@-1(???) e0 preinit fsid 414ba1e0-de90-11ee-95c7-87774f69a715 2024-03-10T03:45:09.867 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 bash[27140]: e6f6d97fafa3d71628a42c4a3e70c0bdbb34c30acc54c464df0282cf620c191d 2024-03-10T03:45:09.867 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 systemd[1]: Started Ceph mon.b for 414ba1e0-de90-11ee-95c7-87774f69a715. 2024-03-10T03:45:09.867 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: mon.b@-1(synchronizing).mds e1 new map 2024-03-10T03:45:09.867 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-10T03:45:09.867 INFO:journalctl@ceph.mon.b.smithi071.stdout: e1 2024-03-10T03:45:09.867 INFO:journalctl@ceph.mon.b.smithi071.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-10T03:45:09.867 INFO:journalctl@ceph.mon.b.smithi071.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-10T03:45:09.867 INFO:journalctl@ceph.mon.b.smithi071.stdout: legacy client fscid: -1 2024-03-10T03:45:09.867 INFO:journalctl@ceph.mon.b.smithi071.stdout: 2024-03-10T03:45:09.867 INFO:journalctl@ceph.mon.b.smithi071.stdout: No filesystems configured 2024-03-10T03:45:09.867 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-10T03:45:09.867 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-10T03:45:09.867 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-10T03:45:09.868 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-10T03:45:09.868 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-10T03:45:09.868 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-10T03:45:09.868 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-10T03:45:09.868 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-10T03:45:09.868 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-10T03:45:09.868 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-10T03:45:09.868 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:09.868 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: Added host smithi071 2024-03-10T03:45:09.868 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:09.868 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:09.868 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:09.868 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:09.868 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:09.869 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi071", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:45:09.869 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:09.869 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:09.869 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: Updating smithi071:/etc/ceph/ceph.conf 2024-03-10T03:45:09.869 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: Updating smithi071:/var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T03:45:09.869 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: Updating smithi071:/etc/ceph/ceph.client.admin.keyring 2024-03-10T03:45:09.869 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: Updating smithi071:/var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/config/ceph.client.admin.keyring 2024-03-10T03:45:09.869 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:09.869 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:09.869 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi071", "caps": []}]: dispatch 2024-03-10T03:45:09.869 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi071", "caps": []}]': finished 2024-03-10T03:45:09.869 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-10T03:45:09.869 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: Deploying daemon agent.smithi071 on smithi071 2024-03-10T03:45:09.869 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:09.869 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:09.870 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:09.870 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:09.870 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:09.870 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:09.870 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:09.870 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:09.870 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:09.870 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:09.870 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:09.870 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:09.870 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:09.870 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:09.870 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:09.870 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:09.870 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/2497286814' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-10T03:45:09.871 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/2497286814' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-10T03:45:09.871 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: osdmap e4: 0 total, 0 up, 0 in 2024-03-10T03:45:09.871 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:09.871 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:09.871 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:09.871 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:09.871 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:09.871 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:09.871 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:09.871 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi006:172.21.15.6=a;smithi006:[v2:172.21.15.6:3301,v1:172.21.15.6:6790]=c;smithi071:172.21.15.71=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:45:09.871 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: Saving service mon spec with placement smithi006:172.21.15.6=a;smithi006:[v2:172.21.15.6:3301,v1:172.21.15.6:6790]=c;smithi071:172.21.15.71=b;count:3 2024-03-10T03:45:09.871 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:09.871 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:09.871 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:09.871 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:09.872 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:09.872 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T03:45:09.872 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:09.872 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: Deploying daemon mon.b on smithi071 2024-03-10T03:45:09.872 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:09.872 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:09.872 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:09.872 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:09 smithi071 ceph-mon[27181]: mon.b@-1(synchronizing).paxosservice(auth 1..5) refresh upgraded, format 0 -> 3 2024-03-10T03:45:11.527 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-10T03:45:11.527 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph mon dump -f json 2024-03-10T03:45:11.771 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T03:45:15.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:14 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T03:45:15.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:14 smithi006 ceph-mon[25256]: mon.a calling monitor election 2024-03-10T03:45:15.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:14 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:45:15.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:14 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:45:15.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:14 smithi006 ceph-mon[25256]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:15.088 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:14 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:45:15.088 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:14 smithi006 ceph-mon[25256]: mon.b calling monitor election 2024-03-10T03:45:15.088 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:14 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:45:15.088 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:14 smithi006 ceph-mon[25256]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:15.088 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:14 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:45:15.088 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:14 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:45:15.089 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:14 smithi006 ceph-mon[25256]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-10T03:45:15.089 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:14 smithi006 ceph-mon[25256]: monmap e2: 2 mons at {a=[v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0],b=[v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-10T03:45:15.089 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:14 smithi006 ceph-mon[25256]: fsmap 2024-03-10T03:45:15.089 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:14 smithi006 ceph-mon[25256]: osdmap e4: 0 total, 0 up, 0 in 2024-03-10T03:45:15.089 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:14 smithi006 ceph-mon[25256]: mgrmap e13: y(active, since 41s) 2024-03-10T03:45:15.089 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:14 smithi006 ceph-mon[25256]: overall HEALTH_OK 2024-03-10T03:45:15.090 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:14 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:15.090 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:14 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:15.090 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:14 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:15.090 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:14 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T03:45:15.090 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:14 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:15.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:14 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T03:45:15.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:14 smithi071 ceph-mon[27181]: mon.a calling monitor election 2024-03-10T03:45:15.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:14 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:45:15.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:14 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:45:15.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:14 smithi071 ceph-mon[27181]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:15.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:14 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:45:15.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:14 smithi071 ceph-mon[27181]: mon.b calling monitor election 2024-03-10T03:45:15.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:14 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:45:15.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:14 smithi071 ceph-mon[27181]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:15.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:14 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:45:15.093 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:14 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:45:15.093 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:14 smithi071 ceph-mon[27181]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-10T03:45:15.093 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:14 smithi071 ceph-mon[27181]: monmap e2: 2 mons at {a=[v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0],b=[v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-10T03:45:15.093 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:14 smithi071 ceph-mon[27181]: fsmap 2024-03-10T03:45:15.093 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:14 smithi071 ceph-mon[27181]: osdmap e4: 0 total, 0 up, 0 in 2024-03-10T03:45:15.093 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:14 smithi071 ceph-mon[27181]: mgrmap e13: y(active, since 41s) 2024-03-10T03:45:15.094 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:14 smithi071 ceph-mon[27181]: overall HEALTH_OK 2024-03-10T03:45:15.094 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:14 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:15.094 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:14 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:15.094 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:14 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:15.094 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:14 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T03:45:15.094 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:14 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:16.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:15 smithi006 ceph-mon[25256]: Deploying daemon mon.c on smithi006 2024-03-10T03:45:16.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:15 smithi006 ceph-mon[25256]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:16.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:15 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:45:16.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:15 smithi071 ceph-mon[27181]: Deploying daemon mon.c on smithi006 2024-03-10T03:45:16.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:15 smithi071 ceph-mon[27181]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:16.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:15 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:45:16.450 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-10T03:45:16.451 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":2,"fsid":"414ba1e0-de90-11ee-95c7-87774f69a715","modified":"2024-03-10T03:45:09.799173Z","created":"2024-03-10T03:43:26.668413Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-10T03:45:16.452 INFO:teuthology.orchestra.run.smithi071.stderr:dumped monmap epoch 2 2024-03-10T03:45:16.997 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:16 smithi071 ceph-mon[27181]: from='client.? 172.21.15.71:0/2587348445' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-10T03:45:17.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:16 smithi006 ceph-mon[25256]: from='client.? 172.21.15.71:0/2587348445' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-10T03:45:18.022 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-10T03:45:18.022 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph mon dump -f json 2024-03-10T03:45:18.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:17 smithi006 ceph-mon[25256]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:18.262 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T03:45:18.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:17 smithi071 ceph-mon[27181]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:18.462 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: load: jerasure load: lrc 2024-03-10T03:45:18.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: RocksDB version: 7.9.2 2024-03-10T03:45:18.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Git sha 0 2024-03-10T03:45:18.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Compile date 2024-03-06 01:10:31 2024-03-10T03:45:18.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: DB SUMMARY 2024-03-10T03:45:18.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: DB Session ID: A77K30ATUSIA2JNC0O80 2024-03-10T03:45:18.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: CURRENT file: CURRENT 2024-03-10T03:45:18.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: IDENTITY file: IDENTITY 2024-03-10T03:45:18.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-10T03:45:18.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-10T03:45:18.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-03-10T03:45:18.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.error_if_exists: 0 2024-03-10T03:45:18.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.create_if_missing: 0 2024-03-10T03:45:18.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.paranoid_checks: 1 2024-03-10T03:45:18.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-10T03:45:18.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-10T03:45:18.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-10T03:45:18.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.env: 0x559874c29b20 2024-03-10T03:45:18.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.fs: PosixFileSystem 2024-03-10T03:45:18.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.info_log: 0x559876bfb8e0 2024-03-10T03:45:18.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_file_opening_threads: 16 2024-03-10T03:45:18.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.statistics: (nil) 2024-03-10T03:45:18.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.use_fsync: 0 2024-03-10T03:45:18.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_log_file_size: 0 2024-03-10T03:45:18.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-10T03:45:18.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-10T03:45:18.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.keep_log_file_num: 1000 2024-03-10T03:45:18.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.recycle_log_file_num: 0 2024-03-10T03:45:18.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.allow_fallocate: 1 2024-03-10T03:45:18.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.allow_mmap_reads: 0 2024-03-10T03:45:18.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.allow_mmap_writes: 0 2024-03-10T03:45:18.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.use_direct_reads: 0 2024-03-10T03:45:18.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-10T03:45:18.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.create_missing_column_families: 0 2024-03-10T03:45:18.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.db_log_dir: 2024-03-10T03:45:18.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.wal_dir: 2024-03-10T03:45:18.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-10T03:45:18.841 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-10T03:45:18.841 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-10T03:45:18.841 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-10T03:45:18.842 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-10T03:45:18.842 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-10T03:45:18.842 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.advise_random_on_open: 1 2024-03-10T03:45:18.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.db_write_buffer_size: 0 2024-03-10T03:45:18.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.write_buffer_manager: 0x559876bcf400 2024-03-10T03:45:18.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-10T03:45:18.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-10T03:45:18.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-10T03:45:18.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.rate_limiter: (nil) 2024-03-10T03:45:18.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-10T03:45:18.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.wal_recovery_mode: 2 2024-03-10T03:45:18.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.enable_thread_tracking: 0 2024-03-10T03:45:18.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.enable_pipelined_write: 0 2024-03-10T03:45:18.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.unordered_write: 0 2024-03-10T03:45:18.845 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-10T03:45:18.845 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-10T03:45:18.845 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-10T03:45:18.845 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-10T03:45:18.845 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.row_cache: None 2024-03-10T03:45:18.845 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.wal_filter: None 2024-03-10T03:45:18.845 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-10T03:45:18.845 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.allow_ingest_behind: 0 2024-03-10T03:45:18.846 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.two_write_queues: 0 2024-03-10T03:45:18.846 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.manual_wal_flush: 0 2024-03-10T03:45:18.846 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.wal_compression: 0 2024-03-10T03:45:18.846 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.atomic_flush: 0 2024-03-10T03:45:18.846 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-10T03:45:18.847 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-10T03:45:18.847 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-10T03:45:18.847 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.log_readahead_size: 0 2024-03-10T03:45:18.847 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-10T03:45:18.847 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.best_efforts_recovery: 0 2024-03-10T03:45:18.847 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-10T03:45:18.847 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-10T03:45:18.847 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.allow_data_in_errors: 0 2024-03-10T03:45:18.847 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.db_host_id: __hostname__ 2024-03-10T03:45:18.848 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-10T03:45:18.848 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_background_jobs: 2 2024-03-10T03:45:18.848 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_background_compactions: -1 2024-03-10T03:45:18.848 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_subcompactions: 1 2024-03-10T03:45:18.848 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-10T03:45:18.848 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-10T03:45:18.848 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-10T03:45:18.848 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_total_wal_size: 0 2024-03-10T03:45:18.848 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-10T03:45:18.848 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-10T03:45:18.848 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-10T03:45:18.848 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-10T03:45:18.849 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_open_files: -1 2024-03-10T03:45:18.849 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.bytes_per_sync: 0 2024-03-10T03:45:18.849 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-10T03:45:18.849 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-10T03:45:18.849 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.compaction_readahead_size: 0 2024-03-10T03:45:18.849 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_background_flushes: -1 2024-03-10T03:45:18.849 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Compression algorithms supported: 2024-03-10T03:45:18.849 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-10T03:45:18.849 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: kZSTD supported: 0 2024-03-10T03:45:18.849 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: kXpressCompression supported: 0 2024-03-10T03:45:18.849 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: kLZ4HCCompression supported: 1 2024-03-10T03:45:18.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: kZlibCompression supported: 1 2024-03-10T03:45:18.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: kSnappyCompression supported: 1 2024-03-10T03:45:18.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: kLZ4Compression supported: 1 2024-03-10T03:45:18.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: kBZip2Compression supported: 0 2024-03-10T03:45:18.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-10T03:45:18.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-10T03:45:18.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-10T03:45:18.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-10T03:45:18.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-10T03:45:18.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.merge_operator: 2024-03-10T03:45:18.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.compaction_filter: None 2024-03-10T03:45:18.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.compaction_filter_factory: None 2024-03-10T03:45:18.851 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.sst_partitioner_factory: None 2024-03-10T03:45:18.851 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-10T03:45:18.851 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-10T03:45:18.851 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559876bfb9e0) 2024-03-10T03:45:18.851 INFO:journalctl@ceph.mon.c.smithi006.stdout: cache_index_and_filter_blocks: 1 2024-03-10T03:45:18.851 INFO:journalctl@ceph.mon.c.smithi006.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-10T03:45:18.851 INFO:journalctl@ceph.mon.c.smithi006.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-10T03:45:18.851 INFO:journalctl@ceph.mon.c.smithi006.stdout: pin_top_level_index_and_filter: 1 2024-03-10T03:45:18.851 INFO:journalctl@ceph.mon.c.smithi006.stdout: index_type: 0 2024-03-10T03:45:18.851 INFO:journalctl@ceph.mon.c.smithi006.stdout: data_block_index_type: 0 2024-03-10T03:45:18.851 INFO:journalctl@ceph.mon.c.smithi006.stdout: index_shortening: 1 2024-03-10T03:45:18.851 INFO:journalctl@ceph.mon.c.smithi006.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-10T03:45:18.851 INFO:journalctl@ceph.mon.c.smithi006.stdout: checksum: 4 2024-03-10T03:45:18.852 INFO:journalctl@ceph.mon.c.smithi006.stdout: no_block_cache: 0 2024-03-10T03:45:18.852 INFO:journalctl@ceph.mon.c.smithi006.stdout: block_cache: 0x559876c01090 2024-03-10T03:45:18.852 INFO:journalctl@ceph.mon.c.smithi006.stdout: block_cache_name: BinnedLRUCache 2024-03-10T03:45:18.852 INFO:journalctl@ceph.mon.c.smithi006.stdout: block_cache_options: 2024-03-10T03:45:18.852 INFO:journalctl@ceph.mon.c.smithi006.stdout: capacity : 536870912 2024-03-10T03:45:18.852 INFO:journalctl@ceph.mon.c.smithi006.stdout: num_shard_bits : 4 2024-03-10T03:45:18.852 INFO:journalctl@ceph.mon.c.smithi006.stdout: strict_capacity_limit : 0 2024-03-10T03:45:18.852 INFO:journalctl@ceph.mon.c.smithi006.stdout: high_pri_pool_ratio: 0.000 2024-03-10T03:45:18.852 INFO:journalctl@ceph.mon.c.smithi006.stdout: block_cache_compressed: (nil) 2024-03-10T03:45:18.852 INFO:journalctl@ceph.mon.c.smithi006.stdout: persistent_cache: (nil) 2024-03-10T03:45:18.852 INFO:journalctl@ceph.mon.c.smithi006.stdout: block_size: 4096 2024-03-10T03:45:18.852 INFO:journalctl@ceph.mon.c.smithi006.stdout: block_size_deviation: 10 2024-03-10T03:45:18.852 INFO:journalctl@ceph.mon.c.smithi006.stdout: block_restart_interval: 16 2024-03-10T03:45:18.853 INFO:journalctl@ceph.mon.c.smithi006.stdout: index_block_restart_interval: 1 2024-03-10T03:45:18.853 INFO:journalctl@ceph.mon.c.smithi006.stdout: metadata_block_size: 4096 2024-03-10T03:45:18.853 INFO:journalctl@ceph.mon.c.smithi006.stdout: partition_filters: 0 2024-03-10T03:45:18.853 INFO:journalctl@ceph.mon.c.smithi006.stdout: use_delta_encoding: 1 2024-03-10T03:45:18.853 INFO:journalctl@ceph.mon.c.smithi006.stdout: filter_policy: bloomfilter 2024-03-10T03:45:18.853 INFO:journalctl@ceph.mon.c.smithi006.stdout: whole_key_filtering: 1 2024-03-10T03:45:18.853 INFO:journalctl@ceph.mon.c.smithi006.stdout: verify_compression: 0 2024-03-10T03:45:18.853 INFO:journalctl@ceph.mon.c.smithi006.stdout: read_amp_bytes_per_bit: 0 2024-03-10T03:45:18.853 INFO:journalctl@ceph.mon.c.smithi006.stdout: format_version: 5 2024-03-10T03:45:18.853 INFO:journalctl@ceph.mon.c.smithi006.stdout: enable_index_compression: 1 2024-03-10T03:45:18.853 INFO:journalctl@ceph.mon.c.smithi006.stdout: block_align: 0 2024-03-10T03:45:18.853 INFO:journalctl@ceph.mon.c.smithi006.stdout: max_auto_readahead_size: 262144 2024-03-10T03:45:18.853 INFO:journalctl@ceph.mon.c.smithi006.stdout: prepopulate_block_cache: 0 2024-03-10T03:45:18.853 INFO:journalctl@ceph.mon.c.smithi006.stdout: initial_auto_readahead_size: 8192 2024-03-10T03:45:18.853 INFO:journalctl@ceph.mon.c.smithi006.stdout: num_file_reads_for_auto_readahead: 2 2024-03-10T03:45:18.854 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.write_buffer_size: 33554432 2024-03-10T03:45:18.854 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_write_buffer_number: 2 2024-03-10T03:45:18.854 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.compression: NoCompression 2024-03-10T03:45:18.854 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.bottommost_compression: Disabled 2024-03-10T03:45:18.854 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.prefix_extractor: nullptr 2024-03-10T03:45:18.854 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-10T03:45:18.854 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.num_levels: 7 2024-03-10T03:45:18.854 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-10T03:45:18.854 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-10T03:45:18.854 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-10T03:45:18.855 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-10T03:45:18.855 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-10T03:45:18.855 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-10T03:45:18.855 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-10T03:45:18.855 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-10T03:45:18.855 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-10T03:45:18.855 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-10T03:45:18.855 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-10T03:45:18.855 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-10T03:45:18.855 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-10T03:45:18.855 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.compression_opts.level: 32767 2024-03-10T03:45:18.855 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.compression_opts.strategy: 0 2024-03-10T03:45:18.856 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-10T03:45:18.856 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-10T03:45:18.856 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-10T03:45:18.856 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-10T03:45:18.856 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.compression_opts.enabled: false 2024-03-10T03:45:18.856 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-10T03:45:18.856 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-10T03:45:18.856 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-10T03:45:18.856 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-10T03:45:18.856 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.target_file_size_base: 67108864 2024-03-10T03:45:18.856 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-10T03:45:18.856 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-10T03:45:18.856 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-10T03:45:18.857 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-10T03:45:18.857 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-10T03:45:18.857 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-10T03:45:18.857 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-10T03:45:18.857 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-10T03:45:18.857 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-10T03:45:18.857 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-10T03:45:18.857 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-10T03:45:18.857 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-10T03:45:18.857 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-10T03:45:18.857 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-10T03:45:18.857 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.arena_block_size: 1048576 2024-03-10T03:45:18.857 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-10T03:45:18.857 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-10T03:45:18.857 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.disable_auto_compactions: 0 2024-03-10T03:45:18.858 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-10T03:45:18.858 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-10T03:45:18.858 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-10T03:45:18.858 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-10T03:45:18.858 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-10T03:45:18.858 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-10T03:45:18.858 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-10T03:45:18.858 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-10T03:45:18.858 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-10T03:45:18.858 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-10T03:45:18.858 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.table_properties_collectors: 2024-03-10T03:45:18.858 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.inplace_update_support: 0 2024-03-10T03:45:18.858 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-10T03:45:18.859 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-10T03:45:18.859 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-10T03:45:18.859 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-10T03:45:18.859 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.bloom_locality: 0 2024-03-10T03:45:18.859 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.max_successive_merges: 0 2024-03-10T03:45:18.859 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-10T03:45:18.859 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.paranoid_file_checks: 0 2024-03-10T03:45:18.859 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.force_consistency_checks: 1 2024-03-10T03:45:18.859 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.report_bg_io_stats: 0 2024-03-10T03:45:18.859 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.ttl: 2592000 2024-03-10T03:45:18.859 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-10T03:45:18.859 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-10T03:45:18.859 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-10T03:45:18.859 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.enable_blob_files: false 2024-03-10T03:45:18.860 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.min_blob_size: 0 2024-03-10T03:45:18.860 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.blob_file_size: 268435456 2024-03-10T03:45:18.860 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-10T03:45:18.860 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-10T03:45:18.860 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-10T03:45:18.860 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-10T03:45:18.860 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-10T03:45:18.860 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.blob_file_starting_level: 0 2024-03-10T03:45:18.860 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-10T03:45:18.860 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-10T03:45:18.860 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-10T03:45:18.860 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 46722a3b-0dbf-4d9f-b6b7-5714b9aae17f 2024-03-10T03:45:18.860 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710042318464638, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-10T03:45:18.860 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-10T03:45:18.861 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710042318465029, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710042318, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "46722a3b-0dbf-4d9f-b6b7-5714b9aae17f", "db_session_id": "A77K30ATUSIA2JNC0O80", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-10T03:45:18.861 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710042318465118, "job": 1, "event": "recovery_finished"} 2024-03-10T03:45:18.861 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-10T03:45:18.861 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-10T03:45:18.861 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559876cf4000 2024-03-10T03:45:18.861 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: DB pointer 0x559876cde000 2024-03-10T03:45:18.861 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-10T03:45:18.861 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-10T03:45:18.861 INFO:journalctl@ceph.mon.c.smithi006.stdout: ** DB Stats ** 2024-03-10T03:45:18.861 INFO:journalctl@ceph.mon.c.smithi006.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-10T03:45:18.861 INFO:journalctl@ceph.mon.c.smithi006.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-10T03:45:18.861 INFO:journalctl@ceph.mon.c.smithi006.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-10T03:45:18.861 INFO:journalctl@ceph.mon.c.smithi006.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-10T03:45:18.861 INFO:journalctl@ceph.mon.c.smithi006.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-10T03:45:18.862 INFO:journalctl@ceph.mon.c.smithi006.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-10T03:45:18.862 INFO:journalctl@ceph.mon.c.smithi006.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-10T03:45:18.862 INFO:journalctl@ceph.mon.c.smithi006.stdout: 2024-03-10T03:45:18.862 INFO:journalctl@ceph.mon.c.smithi006.stdout: ** Compaction Stats [default] ** 2024-03-10T03:45:18.862 INFO:journalctl@ceph.mon.c.smithi006.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-10T03:45:18.862 INFO:journalctl@ceph.mon.c.smithi006.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-10T03:45:18.862 INFO:journalctl@ceph.mon.c.smithi006.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-10T03:45:18.862 INFO:journalctl@ceph.mon.c.smithi006.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-10T03:45:18.862 INFO:journalctl@ceph.mon.c.smithi006.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-10T03:45:18.862 INFO:journalctl@ceph.mon.c.smithi006.stdout: 2024-03-10T03:45:18.862 INFO:journalctl@ceph.mon.c.smithi006.stdout: ** Compaction Stats [default] ** 2024-03-10T03:45:18.862 INFO:journalctl@ceph.mon.c.smithi006.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-10T03:45:18.862 INFO:journalctl@ceph.mon.c.smithi006.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-10T03:45:18.863 INFO:journalctl@ceph.mon.c.smithi006.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-10T03:45:18.863 INFO:journalctl@ceph.mon.c.smithi006.stdout: 2024-03-10T03:45:18.863 INFO:journalctl@ceph.mon.c.smithi006.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-10T03:45:18.863 INFO:journalctl@ceph.mon.c.smithi006.stdout: 2024-03-10T03:45:18.863 INFO:journalctl@ceph.mon.c.smithi006.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-10T03:45:18.863 INFO:journalctl@ceph.mon.c.smithi006.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-10T03:45:18.863 INFO:journalctl@ceph.mon.c.smithi006.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-10T03:45:18.863 INFO:journalctl@ceph.mon.c.smithi006.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-10T03:45:18.863 INFO:journalctl@ceph.mon.c.smithi006.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-10T03:45:18.863 INFO:journalctl@ceph.mon.c.smithi006.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-10T03:45:18.863 INFO:journalctl@ceph.mon.c.smithi006.stdout: Cumulative compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-10T03:45:18.863 INFO:journalctl@ceph.mon.c.smithi006.stdout: Interval compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-10T03:45:18.863 INFO:journalctl@ceph.mon.c.smithi006.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-10T03:45:18.863 INFO:journalctl@ceph.mon.c.smithi006.stdout: Block cache BinnedLRUCache@0x559876c01090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-10T03:45:18.863 INFO:journalctl@ceph.mon.c.smithi006.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-10T03:45:18.864 INFO:journalctl@ceph.mon.c.smithi006.stdout: 2024-03-10T03:45:18.864 INFO:journalctl@ceph.mon.c.smithi006.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-10T03:45:18.864 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-10T03:45:18.864 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: using public_addrv [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2024-03-10T03:45:18.864 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: starting mon.c rank -1 at public addrs [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] at bind addrs [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 414ba1e0-de90-11ee-95c7-87774f69a715 2024-03-10T03:45:18.864 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: mon.c@-1(???) e0 preinit fsid 414ba1e0-de90-11ee-95c7-87774f69a715 2024-03-10T03:45:18.864 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: mon.c@-1(synchronizing).mds e1 new map 2024-03-10T03:45:18.864 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-10T03:45:18.864 INFO:journalctl@ceph.mon.c.smithi006.stdout: e1 2024-03-10T03:45:18.864 INFO:journalctl@ceph.mon.c.smithi006.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-10T03:45:18.864 INFO:journalctl@ceph.mon.c.smithi006.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-10T03:45:18.864 INFO:journalctl@ceph.mon.c.smithi006.stdout: legacy client fscid: -1 2024-03-10T03:45:18.864 INFO:journalctl@ceph.mon.c.smithi006.stdout: 2024-03-10T03:45:18.865 INFO:journalctl@ceph.mon.c.smithi006.stdout: No filesystems configured 2024-03-10T03:45:18.865 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-10T03:45:18.865 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-10T03:45:18.865 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-10T03:45:18.865 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-10T03:45:18.865 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-10T03:45:18.866 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-10T03:45:18.866 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-10T03:45:18.866 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-10T03:45:18.866 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-10T03:45:18.866 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-10T03:45:18.866 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.866 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: Added host smithi071 2024-03-10T03:45:18.866 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:18.866 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.866 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:18.866 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.866 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.866 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi071", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:45:18.866 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:18.867 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:18.867 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: Updating smithi071:/etc/ceph/ceph.conf 2024-03-10T03:45:18.867 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: Updating smithi071:/var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T03:45:18.867 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: Updating smithi071:/etc/ceph/ceph.client.admin.keyring 2024-03-10T03:45:18.867 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: Updating smithi071:/var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/config/ceph.client.admin.keyring 2024-03-10T03:45:18.867 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.867 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.868 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi071", "caps": []}]: dispatch 2024-03-10T03:45:18.868 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi071", "caps": []}]': finished 2024-03-10T03:45:18.868 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-10T03:45:18.868 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: Deploying daemon agent.smithi071 on smithi071 2024-03-10T03:45:18.869 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:18.869 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.869 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.869 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.869 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:18.869 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:18.869 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:18.870 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.870 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:18.870 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:18.870 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:18.870 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.870 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.870 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.870 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.870 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:18.870 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/2497286814' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-10T03:45:18.870 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/2497286814' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-10T03:45:18.871 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: osdmap e4: 0 total, 0 up, 0 in 2024-03-10T03:45:18.871 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:18.871 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.871 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.871 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.871 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.871 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.871 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.871 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi006:172.21.15.6=a;smithi006:[v2:172.21.15.6:3301,v1:172.21.15.6:6790]=c;smithi071:172.21.15.71=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:45:18.871 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: Saving service mon spec with placement smithi006:172.21.15.6=a;smithi006:[v2:172.21.15.6:3301,v1:172.21.15.6:6790]=c;smithi071:172.21.15.71=b;count:3 2024-03-10T03:45:18.871 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.871 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:18.871 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:18.871 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:18.871 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.872 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T03:45:18.872 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:18.872 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: Deploying daemon mon.b on smithi071 2024-03-10T03:45:18.872 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:18.872 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:18.872 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:18.872 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T03:45:18.872 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: mon.a calling monitor election 2024-03-10T03:45:18.872 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:45:18.872 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:45:18.872 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:18.872 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:45:18.872 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: mon.b calling monitor election 2024-03-10T03:45:18.872 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:45:18.873 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:18.873 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:45:18.873 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:45:18.873 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-10T03:45:18.873 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: monmap e2: 2 mons at {a=[v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0],b=[v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-10T03:45:18.873 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: fsmap 2024-03-10T03:45:18.873 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: osdmap e4: 0 total, 0 up, 0 in 2024-03-10T03:45:18.873 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: mgrmap e13: y(active, since 41s) 2024-03-10T03:45:18.873 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: overall HEALTH_OK 2024-03-10T03:45:18.873 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.873 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.873 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:18.873 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T03:45:18.873 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:18.873 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: Deploying daemon mon.c on smithi006 2024-03-10T03:45:18.874 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:18.874 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:45:18.874 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: from='client.? 172.21.15.71:0/2587348445' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-10T03:45:18.874 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:18.874 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 ceph-mon[31842]: mon.c@-1(synchronizing).paxosservice(auth 1..5) refresh upgraded, format 0 -> 3 2024-03-10T03:45:18.874 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 bash[31813]: 2f0215995846b6badac8182726be6361d276aabc35f896fb8cc0f97a90578803 2024-03-10T03:45:18.874 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:18 smithi006 systemd[1]: Started Ceph mon.c for 414ba1e0-de90-11ee-95c7-87774f69a715. 2024-03-10T03:45:23.788 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T03:45:23.789 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: mon.a calling monitor election 2024-03-10T03:45:23.789 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: mon.b calling monitor election 2024-03-10T03:45:23.789 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:45:23.789 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:45:23.789 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:23.789 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:45:23.789 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:45:23.789 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:23.789 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:45:23.789 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:45:23.789 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:23.789 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:45:23.790 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-10T03:45:23.790 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: monmap e3: 3 mons at {a=[v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0],b=[v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0],c=[v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-10T03:45:23.790 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: fsmap 2024-03-10T03:45:23.790 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: osdmap e4: 0 total, 0 up, 0 in 2024-03-10T03:45:23.790 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: mgrmap e13: y(active, since 50s) 2024-03-10T03:45:23.790 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-03-10T03:45:23.790 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-03-10T03:45:23.790 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-03-10T03:45:23.790 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: mon.c (rank 2) addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] is down (out of quorum) 2024-03-10T03:45:23.790 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:23.790 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:23.790 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:23.790 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:23.791 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:23.791 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:23.791 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:23.791 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:23.791 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:23.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T03:45:23.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: mon.a calling monitor election 2024-03-10T03:45:23.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: mon.b calling monitor election 2024-03-10T03:45:23.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:45:23.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:45:23.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:23.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:45:23.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:45:23.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:23.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:45:23.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:45:23.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:23.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:45:23.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-10T03:45:23.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: monmap e3: 3 mons at {a=[v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0],b=[v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0],c=[v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-10T03:45:23.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: fsmap 2024-03-10T03:45:23.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: osdmap e4: 0 total, 0 up, 0 in 2024-03-10T03:45:23.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: mgrmap e13: y(active, since 50s) 2024-03-10T03:45:23.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-03-10T03:45:23.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-03-10T03:45:23.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-03-10T03:45:23.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: mon.c (rank 2) addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] is down (out of quorum) 2024-03-10T03:45:23.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:23.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:23.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:23.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:23.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:23.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:23.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:23.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:23.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:24.464 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-03-10T03:45:24.464 INFO:teuthology.orchestra.run.smithi071.stdout:{"epoch":3,"fsid":"414ba1e0-de90-11ee-95c7-87774f69a715","modified":"2024-03-10T03:45:18.487211Z","created":"2024-03-10T03:43:26.668413Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3301","nonce":0},{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-10T03:45:24.466 INFO:teuthology.orchestra.run.smithi071.stderr:dumped monmap epoch 3 2024-03-10T03:45:24.832 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:24 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:24.832 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:24 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:24.832 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:24 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:24.833 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:24 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:24.833 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:24 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:24.833 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:24 smithi006 ceph-mon[25256]: Updating smithi006:/etc/ceph/ceph.conf 2024-03-10T03:45:24.833 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:24 smithi006 ceph-mon[25256]: Updating smithi071:/etc/ceph/ceph.conf 2024-03-10T03:45:24.833 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:24 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:24.833 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:24 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:24.833 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:24 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:24.833 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:24 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:24.833 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:24 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:24.833 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:24 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:24.833 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:24 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:24.833 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:24 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T03:45:24.833 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:24 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-10T03:45:24.834 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:24 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:24.834 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:24 smithi006 ceph-mon[25256]: from='client.? 172.21.15.71:0/675205693' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-10T03:45:24.834 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:24 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:45:24.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:24 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:24.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:24 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:24.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:24 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:24.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:24 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:24.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:24 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:24.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:24 smithi071 ceph-mon[27181]: Updating smithi006:/etc/ceph/ceph.conf 2024-03-10T03:45:24.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:24 smithi071 ceph-mon[27181]: Updating smithi071:/etc/ceph/ceph.conf 2024-03-10T03:45:24.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:24 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:24.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:24 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:24.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:24 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:24.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:24 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:24.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:24 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:24.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:24 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:24.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:24 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:24.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:24 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T03:45:24.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:24 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-10T03:45:24.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:24 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:24.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:24 smithi071 ceph-mon[27181]: from='client.? 172.21.15.71:0/675205693' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-10T03:45:24.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:24 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:45:25.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T03:45:25.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: mon.a calling monitor election 2024-03-10T03:45:25.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: mon.b calling monitor election 2024-03-10T03:45:25.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:45:25.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:45:25.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:25.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:45:25.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:45:25.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:25.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:45:25.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:45:25.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:25.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:45:25.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-10T03:45:25.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: monmap e3: 3 mons at {a=[v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0],b=[v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0],c=[v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-10T03:45:25.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: fsmap 2024-03-10T03:45:25.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: osdmap e4: 0 total, 0 up, 0 in 2024-03-10T03:45:25.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: mgrmap e13: y(active, since 50s) 2024-03-10T03:45:25.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-03-10T03:45:25.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-03-10T03:45:25.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-03-10T03:45:25.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: mon.c (rank 2) addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] is down (out of quorum) 2024-03-10T03:45:25.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:25.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:25.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:25.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:25.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:25.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:25.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:25.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:25.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:25.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:25.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:25.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:25.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:25.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:25.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: Updating smithi006:/etc/ceph/ceph.conf 2024-03-10T03:45:25.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: Updating smithi071:/etc/ceph/ceph.conf 2024-03-10T03:45:25.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:25.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:25.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:25.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:25.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:25.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:25.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:25.841 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T03:45:25.841 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-10T03:45:25.841 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:25.841 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='client.? 172.21.15.71:0/675205693' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-10T03:45:25.841 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:45:26.116 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-10T03:45:26.116 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph config generate-minimal-conf 2024-03-10T03:45:26.445 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:26 smithi006 ceph-mon[31842]: mon.c calling monitor election 2024-03-10T03:45:26.445 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:26 smithi006 ceph-mon[31842]: mon.c calling monitor election 2024-03-10T03:45:26.445 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:26 smithi006 ceph-mon[31842]: mon.b calling monitor election 2024-03-10T03:45:26.445 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:26 smithi006 ceph-mon[31842]: mon.a calling monitor election 2024-03-10T03:45:26.446 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:26 smithi006 ceph-mon[31842]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-10T03:45:26.446 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:26 smithi006 ceph-mon[31842]: monmap e3: 3 mons at {a=[v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0],b=[v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0],c=[v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-10T03:45:26.446 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:26 smithi006 ceph-mon[31842]: fsmap 2024-03-10T03:45:26.446 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:26 smithi006 ceph-mon[31842]: osdmap e4: 0 total, 0 up, 0 in 2024-03-10T03:45:26.446 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:26 smithi006 ceph-mon[31842]: mgrmap e13: y(active, since 52s) 2024-03-10T03:45:26.446 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:26 smithi006 ceph-mon[31842]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-03-10T03:45:26.446 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:26 smithi006 ceph-mon[31842]: Cluster is now healthy 2024-03-10T03:45:26.446 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:26 smithi006 ceph-mon[31842]: overall HEALTH_OK 2024-03-10T03:45:26.446 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:26 smithi006 ceph-mon[25256]: mon.c calling monitor election 2024-03-10T03:45:26.446 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:26 smithi006 ceph-mon[25256]: mon.c calling monitor election 2024-03-10T03:45:26.446 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:26 smithi006 ceph-mon[25256]: mon.b calling monitor election 2024-03-10T03:45:26.447 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:26 smithi006 ceph-mon[25256]: mon.a calling monitor election 2024-03-10T03:45:26.447 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:26 smithi006 ceph-mon[25256]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-10T03:45:26.447 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:26 smithi006 ceph-mon[25256]: monmap e3: 3 mons at {a=[v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0],b=[v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0],c=[v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-10T03:45:26.447 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:26 smithi006 ceph-mon[25256]: fsmap 2024-03-10T03:45:26.447 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:26 smithi006 ceph-mon[25256]: osdmap e4: 0 total, 0 up, 0 in 2024-03-10T03:45:26.447 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:26 smithi006 ceph-mon[25256]: mgrmap e13: y(active, since 52s) 2024-03-10T03:45:26.447 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:26 smithi006 ceph-mon[25256]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-03-10T03:45:26.447 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:26 smithi006 ceph-mon[25256]: Cluster is now healthy 2024-03-10T03:45:26.447 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:26 smithi006 ceph-mon[25256]: overall HEALTH_OK 2024-03-10T03:45:26.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:26 smithi071 ceph-mon[27181]: mon.c calling monitor election 2024-03-10T03:45:26.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:26 smithi071 ceph-mon[27181]: mon.c calling monitor election 2024-03-10T03:45:26.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:26 smithi071 ceph-mon[27181]: mon.b calling monitor election 2024-03-10T03:45:26.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:26 smithi071 ceph-mon[27181]: mon.a calling monitor election 2024-03-10T03:45:26.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:26 smithi071 ceph-mon[27181]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-10T03:45:26.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:26 smithi071 ceph-mon[27181]: monmap e3: 3 mons at {a=[v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0],b=[v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0],c=[v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-10T03:45:26.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:26 smithi071 ceph-mon[27181]: fsmap 2024-03-10T03:45:26.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:26 smithi071 ceph-mon[27181]: osdmap e4: 0 total, 0 up, 0 in 2024-03-10T03:45:26.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:26 smithi071 ceph-mon[27181]: mgrmap e13: y(active, since 52s) 2024-03-10T03:45:26.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:26 smithi071 ceph-mon[27181]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-03-10T03:45:26.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:26 smithi071 ceph-mon[27181]: Cluster is now healthy 2024-03-10T03:45:26.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:26 smithi071 ceph-mon[27181]: overall HEALTH_OK 2024-03-10T03:45:26.805 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:45:27.459 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:27 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:45:27.459 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:27 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:27.459 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:27 smithi006 ceph-mon[25256]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:27.459 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:27 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:27.459 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:27 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:27.460 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:27 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:27.460 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:27 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:45:27.460 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:27 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:27.460 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:27 smithi006 ceph-mon[31842]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:27.460 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:27 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:27.461 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:27 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:27.461 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:27 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:27.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:27 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:45:27.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:27 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:27.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:27 smithi071 ceph-mon[27181]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:27.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:27 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:27.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:27 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:27.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:27 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:29.047 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:28 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:29.047 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:28 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:29.047 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:28 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:29.048 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:28 smithi006 ceph-mon[31842]: Reconfiguring mon.c (monmap changed)... 2024-03-10T03:45:29.048 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:28 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T03:45:29.048 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:28 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-10T03:45:29.048 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:28 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:29.048 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:28 smithi006 ceph-mon[31842]: Reconfiguring daemon mon.c on smithi006 2024-03-10T03:45:29.048 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:28 smithi006 ceph-mon[31842]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:29.048 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:28 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:29.048 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:28 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:29.048 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:28 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:29.048 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:28 smithi006 ceph-mon[25256]: Reconfiguring mon.c (monmap changed)... 2024-03-10T03:45:29.048 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:28 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T03:45:29.049 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:28 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-10T03:45:29.049 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:28 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:29.049 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:28 smithi006 ceph-mon[25256]: Reconfiguring daemon mon.c on smithi006 2024-03-10T03:45:29.049 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:28 smithi006 ceph-mon[25256]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:29.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:28 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:29.350 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:28 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:29.350 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:28 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:29.350 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:28 smithi071 ceph-mon[27181]: Reconfiguring mon.c (monmap changed)... 2024-03-10T03:45:29.351 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:28 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T03:45:29.351 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:28 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-10T03:45:29.351 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:28 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:29.351 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:28 smithi071 ceph-mon[27181]: Reconfiguring daemon mon.c on smithi006 2024-03-10T03:45:29.351 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:28 smithi071 ceph-mon[27181]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:30.500 INFO:teuthology.orchestra.run.smithi006.stdout:# minimal ceph.conf for 414ba1e0-de90-11ee-95c7-87774f69a715 2024-03-10T03:45:30.500 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-03-10T03:45:30.501 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = 414ba1e0-de90-11ee-95c7-87774f69a715 2024-03-10T03:45:30.501 INFO:teuthology.orchestra.run.smithi006.stdout: mon_host = [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2024-03-10T03:45:31.027 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:30 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/586547706' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:31.027 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:30 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/586547706' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:31.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:30 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/586547706' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:31.964 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:31 smithi006 ceph-mon[31842]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:31.964 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:31 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:31.964 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:31 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:31.964 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:31 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:31.964 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:31 smithi006 ceph-mon[25256]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:31.964 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:31 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:31.964 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:31 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:31.965 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:31 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:32.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:31 smithi071 ceph-mon[27181]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:32.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:31 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:32.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:31 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:32.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:31 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:32.706 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-10T03:45:32.707 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-10T03:45:32.707 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.conf 2024-03-10T03:45:32.738 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-10T03:45:32.738 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-10T03:45:32.807 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-10T03:45:32.807 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.conf 2024-03-10T03:45:32.837 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-10T03:45:32.837 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-10T03:45:32.903 INFO:tasks.cephadm:Adding mgr.y on smithi006 2024-03-10T03:45:32.904 INFO:tasks.cephadm:Adding mgr.x on smithi071 2024-03-10T03:45:32.904 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph orch apply mgr '2;smithi006=y;smithi071=x' 2024-03-10T03:45:33.347 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:33 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:33.347 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:33 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:33.348 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:33 smithi006 ceph-mon[25256]: Reconfiguring mon.b (monmap changed)... 2024-03-10T03:45:33.348 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:33 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T03:45:33.348 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:33 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-10T03:45:33.348 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:33 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:33.348 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:33 smithi006 ceph-mon[25256]: Reconfiguring daemon mon.b on smithi071 2024-03-10T03:45:33.348 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:33 smithi006 ceph-mon[25256]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:33.348 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:33 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:33.348 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:33 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:33.348 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:33 smithi006 ceph-mon[31842]: Reconfiguring mon.b (monmap changed)... 2024-03-10T03:45:33.348 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:33 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T03:45:33.348 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:33 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-10T03:45:33.349 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:33 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:33.349 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:33 smithi006 ceph-mon[31842]: Reconfiguring daemon mon.b on smithi071 2024-03-10T03:45:33.349 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:33 smithi006 ceph-mon[31842]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:33.404 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T03:45:33.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:33 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:33.593 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:33 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:33.593 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:33 smithi071 ceph-mon[27181]: Reconfiguring mon.b (monmap changed)... 2024-03-10T03:45:33.593 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:33 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-10T03:45:33.593 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:33 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-10T03:45:33.593 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:33 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:33.593 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:33 smithi071 ceph-mon[27181]: Reconfiguring daemon mon.b on smithi071 2024-03-10T03:45:33.593 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:33 smithi071 ceph-mon[27181]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:35.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:34 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:35.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:34 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:35.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:34 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:35.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:34 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:35.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:34 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:35.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:34 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:35.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:34 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:35.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:34 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:35.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:34 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:35.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:34 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:35.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:34 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:35.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:34 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:35.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:34 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:35.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:34 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:35.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:34 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:35.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:34 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:35.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:34 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:35.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:34 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:35.642 INFO:teuthology.orchestra.run.smithi071.stdout:Scheduled mgr update... 2024-03-10T03:45:35.951 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:35 smithi071 ceph-mon[27181]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:35.951 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:35 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:35.951 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:35 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:35.951 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:35 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:35.951 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:35 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:35.951 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:35 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:35.951 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:35 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-10T03:45:35.951 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:35 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-10T03:45:35.952 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:35 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-10T03:45:35.952 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:35 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:36.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:35 smithi006 ceph-mon[25256]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:36.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:35 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:36.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:35 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:36.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:35 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:36.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:35 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:36.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:35 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:36.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:35 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-10T03:45:36.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:35 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-10T03:45:36.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:35 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-10T03:45:36.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:35 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:36.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:35 smithi006 ceph-mon[31842]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:36.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:35 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:36.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:35 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:36.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:35 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:36.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:35 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:36.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:35 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:36.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:35 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-10T03:45:36.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:35 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-10T03:45:36.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:35 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-10T03:45:36.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:35 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:36.219 DEBUG:teuthology.orchestra.run.smithi071:mgr.x> sudo journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mgr.x.service 2024-03-10T03:45:36.221 INFO:tasks.cephadm:Deploying OSDs... 2024-03-10T03:45:36.221 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-10T03:45:36.222 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2024-03-10T03:45:36.240 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-10T03:45:36.241 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/nvme1n1 2024-03-10T03:45:36.299 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/nvme1n1 2024-03-10T03:45:36.299 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-10T03:45:36.299 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-10T03:45:36.299 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-10T03:45:36.299 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-10T03:45:36.300 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-10 03:45:30.184949633 +0000 2024-03-10T03:45:36.300 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-10 03:42:42.360590473 +0000 2024-03-10T03:45:36.300 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-10 03:42:42.360590473 +0000 2024-03-10T03:45:36.300 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-10 03:42:42.348590586 +0000 2024-03-10T03:45:36.300 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-10T03:45:36.370 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-10T03:45:36.370 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-10T03:45:36.370 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000298639 s, 1.7 MB/s 2024-03-10T03:45:36.371 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-10T03:45:36.430 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/nvme2n1 2024-03-10T03:45:36.487 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/nvme2n1 2024-03-10T03:45:36.487 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-10T03:45:36.487 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-10T03:45:36.487 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-10T03:45:36.487 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-10T03:45:36.487 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-10 03:45:30.193949548 +0000 2024-03-10T03:45:36.487 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-10 03:42:42.501589141 +0000 2024-03-10T03:45:36.488 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-10 03:42:42.501589141 +0000 2024-03-10T03:45:36.488 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-10 03:42:42.492589226 +0000 2024-03-10T03:45:36.488 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-10T03:45:36.555 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-10T03:45:36.555 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-10T03:45:36.555 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.00021989 s, 2.3 MB/s 2024-03-10T03:45:36.557 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-10T03:45:36.614 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/nvme3n1 2024-03-10T03:45:36.673 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/nvme3n1 2024-03-10T03:45:36.673 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-10T03:45:36.673 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 786 Links: 1 Device type: 103,6 2024-03-10T03:45:36.673 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-10T03:45:36.673 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-10T03:45:36.673 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-10 03:45:30.202949462 +0000 2024-03-10T03:45:36.674 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-10 03:42:42.644587789 +0000 2024-03-10T03:45:36.674 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-10 03:42:42.644587789 +0000 2024-03-10T03:45:36.674 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-10 03:42:42.635587874 +0000 2024-03-10T03:45:36.674 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-10T03:45:36.740 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-10T03:45:36.740 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-10T03:45:36.740 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.00020755 s, 2.5 MB/s 2024-03-10T03:45:36.741 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-10T03:45:36.799 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/nvme4n1 2024-03-10T03:45:36.856 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/nvme4n1 2024-03-10T03:45:36.856 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-10T03:45:36.856 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 798 Links: 1 Device type: 103,8 2024-03-10T03:45:36.856 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-10T03:45:36.856 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-10T03:45:36.856 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-10 03:45:30.211949377 +0000 2024-03-10T03:45:36.856 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-10 03:42:42.787586437 +0000 2024-03-10T03:45:36.857 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-10 03:42:42.787586437 +0000 2024-03-10T03:45:36.857 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-10 03:42:42.778586522 +0000 2024-03-10T03:45:36.857 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-10T03:45:36.923 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-10T03:45:36.923 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-10T03:45:36.923 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000210273 s, 2.4 MB/s 2024-03-10T03:45:36.924 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-10T03:45:36.981 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-10T03:45:36.981 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2024-03-10T03:45:36.998 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-10T03:45:36.998 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/nvme1n1 2024-03-10T03:45:37.052 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/nvme1n1 2024-03-10T03:45:37.052 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-10T03:45:37.052 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-10T03:45:37.052 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-10T03:45:37.052 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-10T03:45:37.052 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-10 03:45:25.919257113 +0000 2024-03-10T03:45:37.052 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-10 03:42:43.970123916 +0000 2024-03-10T03:45:37.052 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-10 03:42:43.970123916 +0000 2024-03-10T03:45:37.052 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-10 03:42:43.957124142 +0000 2024-03-10T03:45:37.053 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-10T03:45:37.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:36 smithi006 ceph-mon[25256]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi006=y;smithi071=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:45:37.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:36 smithi006 ceph-mon[25256]: Saving service mgr spec with placement smithi006=y;smithi071=x;count:2 2024-03-10T03:45:37.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:36 smithi006 ceph-mon[25256]: Deploying daemon mgr.x on smithi071 2024-03-10T03:45:37.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:36 smithi006 ceph-mon[31842]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi006=y;smithi071=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:45:37.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:36 smithi006 ceph-mon[31842]: Saving service mgr spec with placement smithi006=y;smithi071=x;count:2 2024-03-10T03:45:37.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:36 smithi006 ceph-mon[31842]: Deploying daemon mgr.x on smithi071 2024-03-10T03:45:37.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:36 smithi071 ceph-mon[27181]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi006=y;smithi071=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:45:37.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:36 smithi071 ceph-mon[27181]: Saving service mgr spec with placement smithi006=y;smithi071=x;count:2 2024-03-10T03:45:37.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:36 smithi071 ceph-mon[27181]: Deploying daemon mgr.x on smithi071 2024-03-10T03:45:37.118 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-10T03:45:37.118 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-10T03:45:37.118 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.00020836 s, 2.5 MB/s 2024-03-10T03:45:37.119 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-10T03:45:37.174 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/nvme2n1 2024-03-10T03:45:37.228 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/nvme2n1 2024-03-10T03:45:37.229 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-10T03:45:37.229 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 774 Links: 1 Device type: 103,4 2024-03-10T03:45:37.229 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-10T03:45:37.229 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-10T03:45:37.229 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-10 03:45:25.930256920 +0000 2024-03-10T03:45:37.229 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-10 03:42:44.122121279 +0000 2024-03-10T03:45:37.229 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-10 03:42:44.122121279 +0000 2024-03-10T03:45:37.229 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-10 03:42:44.111121470 +0000 2024-03-10T03:45:37.229 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-10T03:45:37.294 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-10T03:45:37.294 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-10T03:45:37.294 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000224602 s, 2.3 MB/s 2024-03-10T03:45:37.295 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-10T03:45:37.350 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/nvme3n1 2024-03-10T03:45:37.406 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/nvme3n1 2024-03-10T03:45:37.406 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-10T03:45:37.406 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-10T03:45:37.406 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-10T03:45:37.406 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-10T03:45:37.406 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-10 03:45:25.941256727 +0000 2024-03-10T03:45:37.406 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-10 03:42:44.274118643 +0000 2024-03-10T03:45:37.406 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-10 03:42:44.274118643 +0000 2024-03-10T03:45:37.406 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-10 03:42:44.263118833 +0000 2024-03-10T03:45:37.407 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-10T03:45:37.594 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-10T03:45:37.594 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-10T03:45:37.595 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000204913 s, 2.5 MB/s 2024-03-10T03:45:37.596 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-10T03:45:37.610 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/nvme4n1 2024-03-10T03:45:37.670 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/nvme4n1 2024-03-10T03:45:37.670 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-10T03:45:37.670 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 798 Links: 1 Device type: 103,8 2024-03-10T03:45:37.670 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-10T03:45:37.670 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-10T03:45:37.670 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-03-10 03:45:25.952256534 +0000 2024-03-10T03:45:37.670 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-03-10 03:42:44.430115937 +0000 2024-03-10T03:45:37.670 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-03-10 03:42:44.430115937 +0000 2024-03-10T03:45:37.670 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-03-10 03:42:44.419116127 +0000 2024-03-10T03:45:37.671 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-10T03:45:37.777 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-03-10T03:45:37.777 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-03-10T03:45:37.777 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000211653 s, 2.4 MB/s 2024-03-10T03:45:37.779 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-10T03:45:37.794 INFO:tasks.cephadm:Deploying osd.0 on smithi006 with /dev/nvme4n1... 2024-03-10T03:45:37.794 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-10T03:45:38.016 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:37 smithi071 ceph-mon[27181]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:38.047 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:45:38.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:37 smithi006 ceph-mon[25256]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:38.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:37 smithi006 ceph-mon[31842]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:38.270 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:38 smithi071 ceph-mgr[29448]: -- 172.21.15.71:0/3361299255 <== mon.1 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e99d1c65a0 con 0x55e99d19b000 2024-03-10T03:45:38.270 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:38 smithi071 bash[29428]: 14276e06b3cbc857d23c9f7756ac06b8ccfbc13109ed149e17f6fccd7daa5a36 2024-03-10T03:45:38.591 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:38 smithi071 systemd[1]: Started Ceph mgr.x for 414ba1e0-de90-11ee-95c7-87774f69a715. 2024-03-10T03:45:39.071 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:38 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:38.759+0000 7f8394fe8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-10T03:45:39.071 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:38 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:38.852+0000 7f8394fe8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-10T03:45:39.379 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:39.379 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:39.379 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:39.379 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:39.379 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:39.379 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:39.379 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:39.379 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:39.379 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[25256]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T03:45:39.379 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[25256]: Reconfiguring mgr.y (unknown last config time)... 2024-03-10T03:45:39.379 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-10T03:45:39.379 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-10T03:45:39.379 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:39.380 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[25256]: Reconfiguring daemon mgr.y on smithi006 2024-03-10T03:45:39.380 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[25256]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:39.380 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:39.380 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:39.380 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:39.380 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:39.380 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:39.380 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:39.380 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:39.380 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:39.380 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[31842]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T03:45:39.380 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[31842]: Reconfiguring mgr.y (unknown last config time)... 2024-03-10T03:45:39.381 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-10T03:45:39.381 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-10T03:45:39.381 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:39.381 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[31842]: Reconfiguring daemon mgr.y on smithi006 2024-03-10T03:45:39.381 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:39 smithi006 ceph-mon[31842]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:39.591 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:39 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:39.515+0000 7f8394fe8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-10T03:45:39.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:39.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:39.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:39.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:39.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:39.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:39.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:39.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:39.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:39 smithi071 ceph-mon[27181]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T03:45:39.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:39 smithi071 ceph-mon[27181]: Reconfiguring mgr.y (unknown last config time)... 2024-03-10T03:45:39.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-10T03:45:39.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-10T03:45:39.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:39.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:39 smithi071 ceph-mon[27181]: Reconfiguring daemon mgr.y on smithi006 2024-03-10T03:45:39.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:39 smithi071 ceph-mon[27181]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:39.855 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:39 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:39.609+0000 7f8394fe8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-10T03:45:39.855 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:39 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:39.759+0000 7f8394fe8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-10T03:45:39.855 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:39 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:39.853+0000 7f8394fe8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-10T03:45:39.946 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping: /dev/nvme4n1 2024-03-10T03:45:39.947 INFO:teuthology.orchestra.run.smithi006.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-10T03:45:39.947 INFO:teuthology.orchestra.run.smithi006.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-10T03:45:39.947 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10+0 records in 2024-03-10T03:45:39.947 INFO:teuthology.orchestra.run.smithi006.stderr:10+0 records out 2024-03-10T03:45:39.947 INFO:teuthology.orchestra.run.smithi006.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0138103 s, 759 MB/s 2024-03-10T03:45:39.947 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping successful for: 2024-03-10T03:45:40.151 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:39 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:39.937+0000 7f8394fe8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-10T03:45:40.420 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:40 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:40.232+0000 7f8394fe8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-10T03:45:40.420 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:40 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:40.345+0000 7f8394fe8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-10T03:45:40.725 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:40 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:40.433+0000 7f8394fe8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-10T03:45:40.725 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:40 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:40.531+0000 7f8394fe8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-10T03:45:40.725 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:40 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:40.723+0000 7f8394fe8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-10T03:45:40.983 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph orch daemon add osd smithi006:/dev/nvme4n1 2024-03-10T03:45:41.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:40 smithi006 ceph-mon[25256]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:41.336 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:40 smithi006 ceph-mon[31842]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:41.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:40 smithi071 ceph-mon[27181]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:41.563 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:45:42.026 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:41 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:41.751+0000 7f8394fe8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-10T03:45:42.026 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:41 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:41.880+0000 7f8394fe8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-10T03:45:42.319 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:42 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:42.229+0000 7f8394fe8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-10T03:45:42.319 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:42 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:42.317+0000 7f8394fe8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-10T03:45:42.591 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:42 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:42.509+0000 7f8394fe8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-10T03:45:42.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:42 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:42 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:42 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:42.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:42 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:42.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:42 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:42.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:42 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:42 smithi071 ceph-mon[27181]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T03:45:42.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:42 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:42 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:42 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:42 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:42.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:42 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:42.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:42 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:42.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:42 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:42 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:42 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:42 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.793 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.793 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.793 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:42.793 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:42.793 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:42.793 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.793 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[31842]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T03:45:42.793 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.793 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.793 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.793 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:42.793 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:42.794 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:42.794 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.794 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.794 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.794 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.794 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.794 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.794 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:42.794 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:42.794 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:42.794 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.794 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[25256]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T03:45:42.795 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.795 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.795 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.795 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:45:42.795 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:42.795 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:45:42.795 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.795 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.795 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.795 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:42 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:42.973 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:42 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:42.691+0000 7f8394fe8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-10T03:45:43.229 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:42 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:42.971+0000 7f8394fe8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-10T03:45:43.230 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:43 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:43.227+0000 7f8394fe8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-10T03:45:43.523 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:43 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:43.459+0000 7f8394fe8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-10T03:45:43.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:43 smithi006 ceph-mon[25256]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:43.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:43.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T03:45:43.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T03:45:43.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:43.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:43 smithi006 ceph-mon[31842]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:43.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:43 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:43.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:43 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T03:45:43.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:43 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T03:45:43.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:43 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:43.840 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:43 smithi071 ceph-mon[27181]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:43.840 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:43 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:43.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:43 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T03:45:43.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:43 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T03:45:43.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:43 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:44.525 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:44.252+0000 7f8394fe8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-10T03:45:44.735 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:44 smithi006 ceph-mon[25256]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:45:44.735 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:44 smithi006 ceph-mon[31842]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:45:44.814 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:44 smithi071 ceph-mon[27181]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:45:45.091 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:44.813+0000 7f8394fe8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-10T03:45:45.091 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:44.910+0000 7f8394fe8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-10T03:45:45.537 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:45:45 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:45:45.121+0000 7f8394fe8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-10T03:45:45.625 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:45 smithi006 ceph-mon[25256]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:45.625 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:45 smithi006 ceph-mon[25256]: Standby manager daemon x started 2024-03-10T03:45:45.625 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:45 smithi006 ceph-mon[25256]: from='mgr.? 172.21.15.71:0/3271385481' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-10T03:45:45.625 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:45 smithi006 ceph-mon[25256]: from='mgr.? 172.21.15.71:0/3271385481' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-10T03:45:45.625 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:45 smithi006 ceph-mon[25256]: from='mgr.? 172.21.15.71:0/3271385481' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-10T03:45:45.625 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:45 smithi006 ceph-mon[25256]: from='mgr.? 172.21.15.71:0/3271385481' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-10T03:45:45.626 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:45 smithi006 ceph-mon[31842]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:45.626 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:45 smithi006 ceph-mon[31842]: Standby manager daemon x started 2024-03-10T03:45:45.626 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:45 smithi006 ceph-mon[31842]: from='mgr.? 172.21.15.71:0/3271385481' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-10T03:45:45.626 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:45 smithi006 ceph-mon[31842]: from='mgr.? 172.21.15.71:0/3271385481' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-10T03:45:45.626 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:45 smithi006 ceph-mon[31842]: from='mgr.? 172.21.15.71:0/3271385481' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-10T03:45:45.626 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:45 smithi006 ceph-mon[31842]: from='mgr.? 172.21.15.71:0/3271385481' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-10T03:45:45.840 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:45 smithi071 ceph-mon[27181]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:45.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:45 smithi071 ceph-mon[27181]: Standby manager daemon x started 2024-03-10T03:45:45.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:45 smithi071 ceph-mon[27181]: from='mgr.? 172.21.15.71:0/3271385481' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-10T03:45:45.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:45 smithi071 ceph-mon[27181]: from='mgr.? 172.21.15.71:0/3271385481' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-10T03:45:45.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:45 smithi071 ceph-mon[27181]: from='mgr.? 172.21.15.71:0/3271385481' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-10T03:45:45.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:45 smithi071 ceph-mon[27181]: from='mgr.? 172.21.15.71:0/3271385481' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-10T03:45:46.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:46 smithi006 ceph-mon[31842]: mgrmap e14: y(active, since 72s), standbys: x 2024-03-10T03:45:46.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:46 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-10T03:45:46.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:46 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/1771852408' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58176ab1-73a2-4d74-8202-a581cd7e934e"}]: dispatch 2024-03-10T03:45:46.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:46 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/1771852408' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58176ab1-73a2-4d74-8202-a581cd7e934e"}]': finished 2024-03-10T03:45:46.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:46 smithi006 ceph-mon[31842]: osdmap e5: 1 total, 0 up, 1 in 2024-03-10T03:45:46.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:46 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T03:45:46.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:46 smithi006 ceph-mon[25256]: mgrmap e14: y(active, since 72s), standbys: x 2024-03-10T03:45:46.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:46 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-10T03:45:46.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:46 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/1771852408' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58176ab1-73a2-4d74-8202-a581cd7e934e"}]: dispatch 2024-03-10T03:45:46.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:46 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/1771852408' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58176ab1-73a2-4d74-8202-a581cd7e934e"}]': finished 2024-03-10T03:45:46.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:46 smithi006 ceph-mon[25256]: osdmap e5: 1 total, 0 up, 1 in 2024-03-10T03:45:46.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:46 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T03:45:46.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:46 smithi071 ceph-mon[27181]: mgrmap e14: y(active, since 72s), standbys: x 2024-03-10T03:45:46.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:46 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-10T03:45:46.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:46 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/1771852408' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58176ab1-73a2-4d74-8202-a581cd7e934e"}]: dispatch 2024-03-10T03:45:46.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:46 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/1771852408' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58176ab1-73a2-4d74-8202-a581cd7e934e"}]': finished 2024-03-10T03:45:46.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:46 smithi071 ceph-mon[27181]: osdmap e5: 1 total, 0 up, 1 in 2024-03-10T03:45:46.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:46 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T03:45:47.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:47 smithi006 ceph-mon[25256]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:47.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:47 smithi006 ceph-mon[31842]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:47.840 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:47 smithi071 ceph-mon[27181]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:48.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:48 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/2391471767' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T03:45:48.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:48 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/2391471767' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T03:45:48.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:48 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/2391471767' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T03:45:49.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:49 smithi006 ceph-mon[25256]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:49.815 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:49 smithi006 ceph-mon[31842]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:49.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:49 smithi071 ceph-mon[27181]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:51.284 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:50 smithi006 ceph-mon[25256]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:51.285 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:50 smithi006 ceph-mon[31842]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:51.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:50 smithi071 ceph-mon[27181]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:52.336 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:52 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:52.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:52 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:52.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:52 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:52.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:52 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:52.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:52 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:52.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:52 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:52.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:52 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:52.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:52 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:52.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:52 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:45:53.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:53 smithi006 ceph-mon[25256]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:53.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:53 smithi006 ceph-mon[31842]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:53.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:53 smithi071 ceph-mon[27181]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:54.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:54 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-10T03:45:54.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:54 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:54.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:54 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-10T03:45:54.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:54 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:54.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:54 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-10T03:45:54.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:54 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:45:55.503 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:55 smithi006 ceph-mon[25256]: Deploying daemon osd.0 on smithi006 2024-03-10T03:45:55.504 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:55 smithi006 ceph-mon[25256]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:55.504 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:55 smithi006 ceph-mon[31842]: Deploying daemon osd.0 on smithi006 2024-03-10T03:45:55.504 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:55 smithi006 ceph-mon[31842]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:55.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:55 smithi071 ceph-mon[27181]: Deploying daemon osd.0 on smithi006 2024-03-10T03:45:55.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:55 smithi071 ceph-mon[27181]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:57.261 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:56 smithi006 ceph-mon[25256]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:57.262 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:56 smithi006 ceph-mon[31842]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:57.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:56 smithi071 ceph-mon[27181]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:59.253 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:45:58 smithi006 ceph-mon[25256]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:59.254 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:45:58 smithi006 ceph-mon[31842]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:45:59.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:45:58 smithi071 ceph-mon[27181]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:46:00.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:00 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:00.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:00 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:00.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:00 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:46:00.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:00 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:00.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:00 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:46:00.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:00 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:00.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:00 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:00.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:00 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:00.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:00 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:00.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:00 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:00.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:00 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:00.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:00 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:46:00.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:00 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:00.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:00 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:46:00.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:00 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:00.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:00 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:00.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:00 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:00.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:00 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:00.840 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:00 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:00.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:00 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:00.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:00 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:46:00.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:00 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:00.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:00 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:46:00.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:00 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:00.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:00 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:00.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:00 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:00.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:00 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:01.832 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:01 smithi006 ceph-mon[25256]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:46:01.833 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:01 smithi006 ceph-mon[31842]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:46:01.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:01 smithi071 ceph-mon[27181]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:46:02.865 INFO:teuthology.orchestra.run.smithi006.stdout:Created osd(s) 0 on host 'smithi006' 2024-03-10T03:46:03.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:02 smithi006 ceph-mon[25256]: from='osd.0 [v2:172.21.15.6:6802/726018156,v1:172.21.15.6:6803/726018156]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-10T03:46:03.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:02 smithi006 ceph-mon[31842]: from='osd.0 [v2:172.21.15.6:6802/726018156,v1:172.21.15.6:6803/726018156]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-10T03:46:03.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:02 smithi071 ceph-mon[27181]: from='osd.0 [v2:172.21.15.6:6802/726018156,v1:172.21.15.6:6803/726018156]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-10T03:46:03.511 DEBUG:teuthology.orchestra.run.smithi006:osd.0> sudo journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.0.service 2024-03-10T03:46:03.514 INFO:tasks.cephadm:Deploying osd.1 on smithi006 with /dev/nvme3n1... 2024-03-10T03:46:03.514 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph-volume lvm zap /dev/nvme3n1 2024-03-10T03:46:03.794 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:46:04.070 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-0[36364]: 2024-03-10T03:46:03.810+0000 7f1453235700 -1 osd.0 0 waiting for initial osdmap 2024-03-10T03:46:04.071 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-0[36364]: 2024-03-10T03:46:03.817+0000 7f144f028700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-10T03:46:04.071 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[25256]: from='osd.0 [v2:172.21.15.6:6802/726018156,v1:172.21.15.6:6803/726018156]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-10T03:46:04.071 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[25256]: osdmap e6: 1 total, 0 up, 1 in 2024-03-10T03:46:04.071 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[25256]: from='osd.0 [v2:172.21.15.6:6802/726018156,v1:172.21.15.6:6803/726018156]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-10T03:46:04.071 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T03:46:04.071 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:04.071 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:04.071 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:46:04.071 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:46:04.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[25256]: Adjusting osd_memory_target on smithi006 to 15045M 2024-03-10T03:46:04.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:04.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:04.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:46:04.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:04.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[25256]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:46:04.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[25256]: from='osd.0 [v2:172.21.15.6:6802/726018156,v1:172.21.15.6:6803/726018156]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-03-10T03:46:04.072 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[31842]: from='osd.0 [v2:172.21.15.6:6802/726018156,v1:172.21.15.6:6803/726018156]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-10T03:46:04.072 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[31842]: osdmap e6: 1 total, 0 up, 1 in 2024-03-10T03:46:04.072 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[31842]: from='osd.0 [v2:172.21.15.6:6802/726018156,v1:172.21.15.6:6803/726018156]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-10T03:46:04.072 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T03:46:04.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:04.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:04.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:46:04.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:46:04.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[31842]: Adjusting osd_memory_target on smithi006 to 15045M 2024-03-10T03:46:04.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:04.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:04.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:46:04.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:04.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[31842]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:46:04.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:03 smithi006 ceph-mon[31842]: from='osd.0 [v2:172.21.15.6:6802/726018156,v1:172.21.15.6:6803/726018156]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-03-10T03:46:04.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:03 smithi071 ceph-mon[27181]: from='osd.0 [v2:172.21.15.6:6802/726018156,v1:172.21.15.6:6803/726018156]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-10T03:46:04.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:03 smithi071 ceph-mon[27181]: osdmap e6: 1 total, 0 up, 1 in 2024-03-10T03:46:04.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:03 smithi071 ceph-mon[27181]: from='osd.0 [v2:172.21.15.6:6802/726018156,v1:172.21.15.6:6803/726018156]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-10T03:46:04.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:03 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T03:46:04.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:03 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:04.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:03 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:04.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:03 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:46:04.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:03 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:46:04.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:03 smithi071 ceph-mon[27181]: Adjusting osd_memory_target on smithi006 to 15045M 2024-03-10T03:46:04.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:03 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:04.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:03 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:04.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:03 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:46:04.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:03 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:04.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:03 smithi071 ceph-mon[27181]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:46:04.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:03 smithi071 ceph-mon[27181]: from='osd.0 [v2:172.21.15.6:6802/726018156,v1:172.21.15.6:6803/726018156]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-03-10T03:46:05.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:04 smithi006 ceph-mon[31842]: osdmap e7: 1 total, 0 up, 1 in 2024-03-10T03:46:05.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:04 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T03:46:05.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:04 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T03:46:05.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:04 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T03:46:05.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:04 smithi006 ceph-mon[25256]: osdmap e7: 1 total, 0 up, 1 in 2024-03-10T03:46:05.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:04 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T03:46:05.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:04 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T03:46:05.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:04 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T03:46:05.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:04 smithi071 ceph-mon[27181]: osdmap e7: 1 total, 0 up, 1 in 2024-03-10T03:46:05.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:04 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T03:46:05.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:04 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T03:46:05.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:04 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T03:46:05.755 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping: /dev/nvme3n1 2024-03-10T03:46:05.755 INFO:teuthology.orchestra.run.smithi006.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-10T03:46:05.755 INFO:teuthology.orchestra.run.smithi006.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme3n1 bs=1M count=10 conv=fsync 2024-03-10T03:46:05.755 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10+0 records in 2024-03-10T03:46:05.755 INFO:teuthology.orchestra.run.smithi006.stderr:10+0 records out 2024-03-10T03:46:05.755 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0132774 s, 790 MB/s 2024-03-10T03:46:05.755 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping successful for: 2024-03-10T03:46:06.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:05 smithi006 ceph-mon[25256]: purged_snaps scrub starts 2024-03-10T03:46:06.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:05 smithi006 ceph-mon[25256]: purged_snaps scrub ok 2024-03-10T03:46:06.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:05 smithi006 ceph-mon[25256]: osd.0 [v2:172.21.15.6:6802/726018156,v1:172.21.15.6:6803/726018156] boot 2024-03-10T03:46:06.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:05 smithi006 ceph-mon[25256]: osdmap e8: 1 total, 1 up, 1 in 2024-03-10T03:46:06.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:05 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T03:46:06.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:05 smithi006 ceph-mon[25256]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:46:06.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:05 smithi006 ceph-mon[31842]: purged_snaps scrub starts 2024-03-10T03:46:06.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:05 smithi006 ceph-mon[31842]: purged_snaps scrub ok 2024-03-10T03:46:06.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:05 smithi006 ceph-mon[31842]: osd.0 [v2:172.21.15.6:6802/726018156,v1:172.21.15.6:6803/726018156] boot 2024-03-10T03:46:06.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:05 smithi006 ceph-mon[31842]: osdmap e8: 1 total, 1 up, 1 in 2024-03-10T03:46:06.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:05 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T03:46:06.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:05 smithi006 ceph-mon[31842]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:46:06.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:05 smithi071 ceph-mon[27181]: purged_snaps scrub starts 2024-03-10T03:46:06.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:05 smithi071 ceph-mon[27181]: purged_snaps scrub ok 2024-03-10T03:46:06.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:05 smithi071 ceph-mon[27181]: osd.0 [v2:172.21.15.6:6802/726018156,v1:172.21.15.6:6803/726018156] boot 2024-03-10T03:46:06.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:05 smithi071 ceph-mon[27181]: osdmap e8: 1 total, 1 up, 1 in 2024-03-10T03:46:06.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:05 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T03:46:06.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:05 smithi071 ceph-mon[27181]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-10T03:46:06.446 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph orch daemon add osd smithi006:/dev/nvme3n1 2024-03-10T03:46:07.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:06 smithi006 ceph-mon[25256]: osdmap e9: 1 total, 1 up, 1 in 2024-03-10T03:46:07.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:06 smithi006 ceph-mon[31842]: osdmap e9: 1 total, 1 up, 1 in 2024-03-10T03:46:07.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:06 smithi071 ceph-mon[27181]: osdmap e9: 1 total, 1 up, 1 in 2024-03-10T03:46:07.234 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:46:08.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:07 smithi006 ceph-mon[25256]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:08.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:07 smithi006 ceph-mon[31842]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:08.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:07 smithi071 ceph-mon[27181]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:09.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:09 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:09.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:09 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:09.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:09 smithi006 ceph-mon[25256]: Detected new or changed devices on smithi006 2024-03-10T03:46:09.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:09 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:09.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:09 smithi006 ceph-mon[25256]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:09.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:09 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T03:46:09.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:09 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T03:46:09.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:09 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:09.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:09 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:09.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:09 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:09.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:09 smithi006 ceph-mon[31842]: Detected new or changed devices on smithi006 2024-03-10T03:46:09.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:09 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:09.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:09 smithi006 ceph-mon[31842]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:09.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:09 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T03:46:09.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:09 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T03:46:09.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:09 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:09.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:09.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:09.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:09 smithi071 ceph-mon[27181]: Detected new or changed devices on smithi006 2024-03-10T03:46:09.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:09.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:09 smithi071 ceph-mon[27181]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:09.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T03:46:09.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T03:46:09.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:10.634 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:10 smithi006 ceph-mon[25256]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:46:10.634 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:10 smithi006 ceph-mon[31842]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:46:10.840 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:10 smithi071 ceph-mon[27181]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:46:11.674 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:11 smithi006 ceph-mon[25256]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:11.674 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:11 smithi006 ceph-mon[31842]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:11.840 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:11 smithi071 ceph-mon[27181]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:13.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:12 smithi006 ceph-mon[25256]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:13.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:12 smithi006 ceph-mon[31842]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:13.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:12 smithi071 ceph-mon[27181]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:14.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:13 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/2999512082' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4eee59a-8e2b-4d51-8f0e-feb6600222b4"}]: dispatch 2024-03-10T03:46:14.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:13 smithi006 ceph-mon[25256]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4eee59a-8e2b-4d51-8f0e-feb6600222b4"}]: dispatch 2024-03-10T03:46:14.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:13 smithi006 ceph-mon[25256]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4eee59a-8e2b-4d51-8f0e-feb6600222b4"}]': finished 2024-03-10T03:46:14.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:13 smithi006 ceph-mon[25256]: osdmap e10: 2 total, 1 up, 2 in 2024-03-10T03:46:14.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:13 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T03:46:14.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:13 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/2999512082' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4eee59a-8e2b-4d51-8f0e-feb6600222b4"}]: dispatch 2024-03-10T03:46:14.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:13 smithi006 ceph-mon[31842]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4eee59a-8e2b-4d51-8f0e-feb6600222b4"}]: dispatch 2024-03-10T03:46:14.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:13 smithi006 ceph-mon[31842]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4eee59a-8e2b-4d51-8f0e-feb6600222b4"}]': finished 2024-03-10T03:46:14.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:13 smithi006 ceph-mon[31842]: osdmap e10: 2 total, 1 up, 2 in 2024-03-10T03:46:14.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:13 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T03:46:14.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:13 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/2999512082' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4eee59a-8e2b-4d51-8f0e-feb6600222b4"}]: dispatch 2024-03-10T03:46:14.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:13 smithi071 ceph-mon[27181]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4eee59a-8e2b-4d51-8f0e-feb6600222b4"}]: dispatch 2024-03-10T03:46:14.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:13 smithi071 ceph-mon[27181]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4eee59a-8e2b-4d51-8f0e-feb6600222b4"}]': finished 2024-03-10T03:46:14.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:13 smithi071 ceph-mon[27181]: osdmap e10: 2 total, 1 up, 2 in 2024-03-10T03:46:14.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:13 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T03:46:15.295 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:15 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/2540887498' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T03:46:15.295 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:15 smithi006 ceph-mon[25256]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:15.295 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:15 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/2540887498' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T03:46:15.295 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:15 smithi006 ceph-mon[31842]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:15.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:15 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/2540887498' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T03:46:15.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:15 smithi071 ceph-mon[27181]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:17.063 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:16 smithi006 ceph-mon[25256]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:17.063 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:16 smithi006 ceph-mon[31842]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:17.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:16 smithi071 ceph-mon[27181]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:19.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:18 smithi006 ceph-mon[25256]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:19.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:18 smithi006 ceph-mon[31842]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:19.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:18 smithi071 ceph-mon[27181]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:21.349 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:21.349 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:21.349 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:21.349 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-10T03:46:21.349 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:21.350 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:21 smithi071 ceph-mon[27181]: Deploying daemon osd.1 on smithi006 2024-03-10T03:46:21.350 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:21 smithi071 ceph-mon[27181]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:21.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:21.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:21.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:21.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-10T03:46:21.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:21.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:21 smithi006 ceph-mon[25256]: Deploying daemon osd.1 on smithi006 2024-03-10T03:46:21.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:21 smithi006 ceph-mon[25256]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:21.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:21.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:21.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:21.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-10T03:46:21.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:21.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:21 smithi006 ceph-mon[31842]: Deploying daemon osd.1 on smithi006 2024-03-10T03:46:21.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:21 smithi006 ceph-mon[31842]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:23.214 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:22 smithi006 ceph-mon[25256]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:23.215 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:22 smithi006 ceph-mon[31842]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:23.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:22 smithi071 ceph-mon[27181]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:25.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:24 smithi006 ceph-mon[25256]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:25.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:24 smithi006 ceph-mon[31842]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:25.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:24 smithi071 ceph-mon[27181]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:27.266 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:26 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:27.266 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:26 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:27.266 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:26 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:46:27.266 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:26 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:27.266 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:26 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:46:27.266 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:26 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:27.266 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:26 smithi006 ceph-mon[31842]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:27.266 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:26 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:27.267 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:26 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:27.267 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:26 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:46:27.267 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:26 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:27.267 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:26 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:46:27.267 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:26 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:27.267 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:26 smithi006 ceph-mon[25256]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:27.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:26 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:27.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:26 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:27.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:26 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:46:27.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:26 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:27.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:26 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:46:27.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:26 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:27.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:26 smithi071 ceph-mon[27181]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:30.005 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:29 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:30.005 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:29 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:30.005 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:29 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:30.005 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:29 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:46:30.005 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:29 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:30.005 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:29 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:46:30.005 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:29 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:30.006 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:29 smithi006 ceph-mon[25256]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:30.006 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:29 smithi006 ceph-mon[25256]: from='osd.1 [v2:172.21.15.6:6810/399164807,v1:172.21.15.6:6811/399164807]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-10T03:46:30.006 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:29 smithi006 ceph-mon[25256]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-10T03:46:30.006 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:29 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:30.006 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:29 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:30.006 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:29 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:30.006 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:29 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:46:30.006 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:29 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:30.006 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:29 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:46:30.006 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:29 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:30.007 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:29 smithi006 ceph-mon[31842]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:30.007 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:29 smithi006 ceph-mon[31842]: from='osd.1 [v2:172.21.15.6:6810/399164807,v1:172.21.15.6:6811/399164807]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-10T03:46:30.007 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:29 smithi006 ceph-mon[31842]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-10T03:46:30.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:29 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:30.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:29 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:30.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:29 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:30.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:29 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:46:30.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:29 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:30.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:29 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:46:30.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:29 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:30.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:29 smithi071 ceph-mon[27181]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:30.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:29 smithi071 ceph-mon[27181]: from='osd.1 [v2:172.21.15.6:6810/399164807,v1:172.21.15.6:6811/399164807]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-10T03:46:30.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:29 smithi071 ceph-mon[27181]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-10T03:46:30.094 INFO:teuthology.orchestra.run.smithi006.stdout:Created osd(s) 1 on host 'smithi006' 2024-03-10T03:46:30.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:30 smithi006 ceph-mon[25256]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-10T03:46:30.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:30 smithi006 ceph-mon[25256]: osdmap e11: 2 total, 1 up, 2 in 2024-03-10T03:46:30.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:30 smithi006 ceph-mon[25256]: from='osd.1 [v2:172.21.15.6:6810/399164807,v1:172.21.15.6:6811/399164807]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-10T03:46:30.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:30 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T03:46:30.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:30 smithi006 ceph-mon[25256]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-10T03:46:30.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:30 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:30.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:30 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:30.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:30 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:46:30.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:30 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:30.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:30 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:46:30.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:30 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:30.967 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:30 smithi006 ceph-mon[31842]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-10T03:46:30.967 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:30 smithi006 ceph-mon[31842]: osdmap e11: 2 total, 1 up, 2 in 2024-03-10T03:46:30.967 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:30 smithi006 ceph-mon[31842]: from='osd.1 [v2:172.21.15.6:6810/399164807,v1:172.21.15.6:6811/399164807]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-10T03:46:30.967 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:30 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T03:46:30.967 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:30 smithi006 ceph-mon[31842]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-10T03:46:30.967 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:30 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:30.967 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:30 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:30.968 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:30 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:46:30.968 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:30 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:30.968 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:30 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:46:30.968 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:30 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:31.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:30 smithi071 ceph-mon[27181]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-10T03:46:31.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:30 smithi071 ceph-mon[27181]: osdmap e11: 2 total, 1 up, 2 in 2024-03-10T03:46:31.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:30 smithi071 ceph-mon[27181]: from='osd.1 [v2:172.21.15.6:6810/399164807,v1:172.21.15.6:6811/399164807]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-10T03:46:31.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:30 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T03:46:31.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:30 smithi071 ceph-mon[27181]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-10T03:46:31.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:30 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:31.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:30 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:31.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:30 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:46:31.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:30 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:31.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:30 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:46:31.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:30 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:31.902 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:31 smithi006 ceph-mon[25256]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-03-10T03:46:31.902 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:31 smithi006 ceph-mon[25256]: osdmap e12: 2 total, 1 up, 2 in 2024-03-10T03:46:31.902 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:31 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T03:46:31.902 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:31 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T03:46:31.903 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:31 smithi006 ceph-mon[25256]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:31.903 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:31 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T03:46:31.903 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:31 smithi006 ceph-mon[31842]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-03-10T03:46:31.903 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:31 smithi006 ceph-mon[31842]: osdmap e12: 2 total, 1 up, 2 in 2024-03-10T03:46:31.903 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:31 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T03:46:31.903 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:31 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T03:46:31.903 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:31 smithi006 ceph-mon[31842]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:31.903 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:31 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T03:46:31.904 DEBUG:teuthology.orchestra.run.smithi006:osd.1> sudo journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.1.service 2024-03-10T03:46:31.906 INFO:tasks.cephadm:Deploying osd.2 on smithi006 with /dev/nvme2n1... 2024-03-10T03:46:31.907 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph-volume lvm zap /dev/nvme2n1 2024-03-10T03:46:32.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:31 smithi071 ceph-mon[27181]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-03-10T03:46:32.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:31 smithi071 ceph-mon[27181]: osdmap e12: 2 total, 1 up, 2 in 2024-03-10T03:46:32.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:31 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T03:46:32.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:31 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T03:46:32.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:31 smithi071 ceph-mon[27181]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-10T03:46:32.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:31 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T03:46:32.672 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:46:32.816 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:32 smithi006 ceph-mon[25256]: purged_snaps scrub starts 2024-03-10T03:46:32.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:32 smithi006 ceph-mon[25256]: purged_snaps scrub ok 2024-03-10T03:46:32.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:32 smithi006 ceph-mon[25256]: osd.1 [v2:172.21.15.6:6810/399164807,v1:172.21.15.6:6811/399164807] boot 2024-03-10T03:46:32.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:32 smithi006 ceph-mon[25256]: osdmap e13: 2 total, 2 up, 2 in 2024-03-10T03:46:32.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:32 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T03:46:32.817 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:32 smithi006 ceph-mon[31842]: purged_snaps scrub starts 2024-03-10T03:46:32.817 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:32 smithi006 ceph-mon[31842]: purged_snaps scrub ok 2024-03-10T03:46:32.817 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:32 smithi006 ceph-mon[31842]: osd.1 [v2:172.21.15.6:6810/399164807,v1:172.21.15.6:6811/399164807] boot 2024-03-10T03:46:32.817 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:32 smithi006 ceph-mon[31842]: osdmap e13: 2 total, 2 up, 2 in 2024-03-10T03:46:32.817 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:32 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T03:46:33.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:32 smithi071 ceph-mon[27181]: purged_snaps scrub starts 2024-03-10T03:46:33.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:32 smithi071 ceph-mon[27181]: purged_snaps scrub ok 2024-03-10T03:46:33.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:32 smithi071 ceph-mon[27181]: osd.1 [v2:172.21.15.6:6810/399164807,v1:172.21.15.6:6811/399164807] boot 2024-03-10T03:46:33.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:32 smithi071 ceph-mon[27181]: osdmap e13: 2 total, 2 up, 2 in 2024-03-10T03:46:33.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:32 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T03:46:33.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:33 smithi006 ceph-mon[25256]: osdmap e14: 2 total, 2 up, 2 in 2024-03-10T03:46:33.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:33 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:33.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:33 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:33.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:33 smithi006 ceph-mon[25256]: Detected new or changed devices on smithi006 2024-03-10T03:46:33.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:33 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:33.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:33 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:46:33.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:33 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:33.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:33 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:46:33.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:33 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:33.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:33 smithi006 ceph-mon[25256]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:33.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:33 smithi006 ceph-mon[31842]: osdmap e14: 2 total, 2 up, 2 in 2024-03-10T03:46:33.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:33 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:33.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:33 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:33.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:33 smithi006 ceph-mon[31842]: Detected new or changed devices on smithi006 2024-03-10T03:46:33.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:33 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:33.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:33 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:46:33.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:33 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:33.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:33 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:46:33.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:33 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:33.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:33 smithi006 ceph-mon[31842]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:34.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:33 smithi071 ceph-mon[27181]: osdmap e14: 2 total, 2 up, 2 in 2024-03-10T03:46:34.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:33 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:34.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:33 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:34.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:33 smithi071 ceph-mon[27181]: Detected new or changed devices on smithi006 2024-03-10T03:46:34.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:33 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:34.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:33 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:46:34.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:33 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:34.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:33 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:46:34.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:33 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:34.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:33 smithi071 ceph-mon[27181]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:34.648 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping: /dev/nvme2n1 2024-03-10T03:46:34.649 INFO:teuthology.orchestra.run.smithi006.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-10T03:46:34.649 INFO:teuthology.orchestra.run.smithi006.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme2n1 bs=1M count=10 conv=fsync 2024-03-10T03:46:34.649 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10+0 records in 2024-03-10T03:46:34.649 INFO:teuthology.orchestra.run.smithi006.stderr:10+0 records out 2024-03-10T03:46:34.649 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123369 s, 850 MB/s 2024-03-10T03:46:34.649 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping successful for: 2024-03-10T03:46:35.326 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:34 smithi006 ceph-mon[25256]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:35.326 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:34 smithi006 ceph-mon[31842]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:35.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:34 smithi071 ceph-mon[27181]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:35.353 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph orch daemon add osd smithi006:/dev/nvme2n1 2024-03-10T03:46:35.599 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:46:37.211 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:37 smithi006 ceph-mon[25256]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:37.212 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:37 smithi006 ceph-mon[31842]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:37.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:37 smithi071 ceph-mon[27181]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:38.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:38 smithi006 ceph-mon[25256]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:46:38.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:38 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T03:46:38.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:38 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T03:46:38.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:38 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:38.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:38 smithi006 ceph-mon[31842]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:46:38.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:38 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T03:46:38.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:38 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T03:46:38.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:38 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:38.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:38 smithi071 ceph-mon[27181]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:46:38.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:38 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T03:46:38.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:38 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T03:46:38.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:38 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:39.129 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:39 smithi006 ceph-mon[25256]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:39.129 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:39 smithi006 ceph-mon[31842]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:39.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:39 smithi071 ceph-mon[27181]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:40.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:40 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/3465073084' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eadadcfe-1185-4ea7-9eee-cb2c346b70b2"}]: dispatch 2024-03-10T03:46:40.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:40 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/3465073084' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eadadcfe-1185-4ea7-9eee-cb2c346b70b2"}]': finished 2024-03-10T03:46:40.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:40 smithi006 ceph-mon[25256]: osdmap e15: 3 total, 2 up, 3 in 2024-03-10T03:46:40.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:40 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T03:46:40.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:40 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/3465073084' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eadadcfe-1185-4ea7-9eee-cb2c346b70b2"}]: dispatch 2024-03-10T03:46:40.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:40 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/3465073084' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eadadcfe-1185-4ea7-9eee-cb2c346b70b2"}]': finished 2024-03-10T03:46:40.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:40 smithi006 ceph-mon[31842]: osdmap e15: 3 total, 2 up, 3 in 2024-03-10T03:46:40.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:40 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T03:46:40.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:40 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/3465073084' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eadadcfe-1185-4ea7-9eee-cb2c346b70b2"}]: dispatch 2024-03-10T03:46:40.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:40 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/3465073084' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eadadcfe-1185-4ea7-9eee-cb2c346b70b2"}]': finished 2024-03-10T03:46:40.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:40 smithi071 ceph-mon[27181]: osdmap e15: 3 total, 2 up, 3 in 2024-03-10T03:46:40.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:40 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T03:46:41.571 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:41 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:41.572 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:41 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:41.572 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:41 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:41.572 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:41 smithi071 ceph-mon[27181]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:41.572 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:41 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/3170247473' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T03:46:41.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:41 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:41.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:41 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:41.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:41 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:41.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:41 smithi006 ceph-mon[25256]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:41.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:41 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/3170247473' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T03:46:41.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:41 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:41.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:41 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:41.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:41 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:41.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:41 smithi006 ceph-mon[31842]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:41.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:41 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/3170247473' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T03:46:43.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:43 smithi006 ceph-mon[25256]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:43.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:43 smithi006 ceph-mon[31842]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:43.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:43 smithi071 ceph-mon[27181]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:45.048 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:45 smithi006 ceph-mon[25256]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:45.048 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:45 smithi006 ceph-mon[31842]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:45.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:45 smithi071 ceph-mon[27181]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:47.200 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:47 smithi006 ceph-mon[25256]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:47.201 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:47 smithi006 ceph-mon[31842]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:47.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:47 smithi071 ceph-mon[27181]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:48.178 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:48 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-10T03:46:48.178 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:48 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:48.178 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:48 smithi006 ceph-mon[31842]: Deploying daemon osd.2 on smithi006 2024-03-10T03:46:48.179 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:48 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-10T03:46:48.179 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:48 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:48.179 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:48 smithi006 ceph-mon[25256]: Deploying daemon osd.2 on smithi006 2024-03-10T03:46:48.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:48 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-10T03:46:48.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:48 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:48.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:48 smithi071 ceph-mon[27181]: Deploying daemon osd.2 on smithi006 2024-03-10T03:46:49.177 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:49 smithi006 ceph-mon[25256]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:49.177 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:49 smithi006 ceph-mon[31842]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:49.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:49 smithi071 ceph-mon[27181]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:51.088 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:51 smithi006 ceph-mon[25256]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:51.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:51 smithi006 ceph-mon[31842]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:51.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:51 smithi071 ceph-mon[27181]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:54.205 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:53 smithi006 ceph-mon[25256]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:54.205 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:53 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:54.205 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:53 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:54.205 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:53 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:54.205 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:53 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:54.206 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:53 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:46:54.206 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:53 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:54.206 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:53 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:54.206 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:53 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:46:54.206 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:53 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:54.206 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:53 smithi006 ceph-mon[31842]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:54.206 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:53 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:54.206 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:53 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:54.206 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:53 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:54.206 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:53 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:54.206 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:53 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:46:54.207 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:53 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:54.207 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:53 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:54.207 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:53 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:46:54.207 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:53 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:54.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:53 smithi071 ceph-mon[27181]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:54.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:53 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:54.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:53 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:54.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:53 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:54.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:53 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:54.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:53 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:46:54.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:53 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:54.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:53 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:54.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:53 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:46:54.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:53 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:56.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:55 smithi006 ceph-mon[25256]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:56.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:55 smithi006 ceph-mon[31842]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:56.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:55 smithi071 ceph-mon[27181]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:57.731 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:57 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:57.731 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:57 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:57.731 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:57 smithi006 ceph-mon[25256]: Detected new or changed devices on smithi006 2024-03-10T03:46:57.731 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:57 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:57.731 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:57 smithi006 ceph-mon[25256]: from='osd.2 [v2:172.21.15.6:6818/3078148920,v1:172.21.15.6:6819/3078148920]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-10T03:46:57.731 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:57 smithi006 ceph-mon[25256]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:57.732 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:57 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:57.732 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:57 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:57.732 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:57 smithi006 ceph-mon[31842]: Detected new or changed devices on smithi006 2024-03-10T03:46:57.732 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:57 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:57.732 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:57 smithi006 ceph-mon[31842]: from='osd.2 [v2:172.21.15.6:6818/3078148920,v1:172.21.15.6:6819/3078148920]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-10T03:46:57.732 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:57 smithi006 ceph-mon[31842]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:57.774 INFO:teuthology.orchestra.run.smithi006.stdout:Created osd(s) 2 on host 'smithi006' 2024-03-10T03:46:57.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:57 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:57.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:57 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:57.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:57 smithi071 ceph-mon[27181]: Detected new or changed devices on smithi006 2024-03-10T03:46:57.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:57 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:57.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:57 smithi071 ceph-mon[27181]: from='osd.2 [v2:172.21.15.6:6818/3078148920,v1:172.21.15.6:6819/3078148920]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-10T03:46:57.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:57 smithi071 ceph-mon[27181]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:58.360 DEBUG:teuthology.orchestra.run.smithi006:osd.2> sudo journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.2.service 2024-03-10T03:46:58.363 INFO:tasks.cephadm:Deploying osd.3 on smithi006 with /dev/nvme1n1... 2024-03-10T03:46:58.363 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph-volume lvm zap /dev/nvme1n1 2024-03-10T03:46:58.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:58 smithi071 ceph-mon[27181]: from='osd.2 [v2:172.21.15.6:6818/3078148920,v1:172.21.15.6:6819/3078148920]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-10T03:46:58.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:58 smithi071 ceph-mon[27181]: osdmap e16: 3 total, 2 up, 3 in 2024-03-10T03:46:58.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:58 smithi071 ceph-mon[27181]: from='osd.2 [v2:172.21.15.6:6818/3078148920,v1:172.21.15.6:6819/3078148920]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-10T03:46:58.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:58 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T03:46:58.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:58 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:58.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:58 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:58.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:58 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:46:58.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:58 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:46:58.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:58 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:46:58.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:58 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:46:58.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:58 smithi071 ceph-mon[27181]: Adjusting osd_memory_target on smithi006 to 5015M 2024-03-10T03:46:58.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:58 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:58.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:58 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:58.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:58 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:46:58.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:58 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:58.650 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:46:58.769 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[25256]: from='osd.2 [v2:172.21.15.6:6818/3078148920,v1:172.21.15.6:6819/3078148920]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-10T03:46:58.769 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[25256]: osdmap e16: 3 total, 2 up, 3 in 2024-03-10T03:46:58.769 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[25256]: from='osd.2 [v2:172.21.15.6:6818/3078148920,v1:172.21.15.6:6819/3078148920]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-10T03:46:58.769 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T03:46:58.769 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:58.769 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:58.769 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:46:58.769 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:46:58.769 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:46:58.769 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:46:58.769 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[25256]: Adjusting osd_memory_target on smithi006 to 5015M 2024-03-10T03:46:58.770 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:58.770 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:58.770 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:46:58.770 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:58.770 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[31842]: from='osd.2 [v2:172.21.15.6:6818/3078148920,v1:172.21.15.6:6819/3078148920]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-10T03:46:58.770 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[31842]: osdmap e16: 3 total, 2 up, 3 in 2024-03-10T03:46:58.770 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[31842]: from='osd.2 [v2:172.21.15.6:6818/3078148920,v1:172.21.15.6:6819/3078148920]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-10T03:46:58.770 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T03:46:58.770 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:58.770 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:58.770 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:46:58.770 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:46:58.771 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:46:58.771 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:46:58.771 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[31842]: Adjusting osd_memory_target on smithi006 to 5015M 2024-03-10T03:46:58.771 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:58.771 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:46:58.771 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:46:58.771 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:46:59.087 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-2[43581]: 2024-03-10T03:46:58.831+0000 7fa1ea6f1700 -1 osd.2 0 waiting for initial osdmap 2024-03-10T03:46:59.087 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 10 03:46:58 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-2[43581]: 2024-03-10T03:46:58.834+0000 7fa1e1cdb700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-10T03:46:59.699 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:59 smithi006 ceph-mon[25256]: from='osd.2 [v2:172.21.15.6:6818/3078148920,v1:172.21.15.6:6819/3078148920]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-03-10T03:46:59.699 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:59 smithi006 ceph-mon[25256]: osdmap e17: 3 total, 2 up, 3 in 2024-03-10T03:46:59.699 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:59 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T03:46:59.699 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:59 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T03:46:59.699 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:59 smithi006 ceph-mon[25256]: from='osd.2 [v2:172.21.15.6:6818/3078148920,v1:172.21.15.6:6819/3078148920]' entity='osd.2' 2024-03-10T03:46:59.699 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:46:59 smithi006 ceph-mon[25256]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:59.699 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:59 smithi006 ceph-mon[31842]: from='osd.2 [v2:172.21.15.6:6818/3078148920,v1:172.21.15.6:6819/3078148920]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-03-10T03:46:59.699 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:59 smithi006 ceph-mon[31842]: osdmap e17: 3 total, 2 up, 3 in 2024-03-10T03:46:59.699 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:59 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T03:46:59.700 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:59 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T03:46:59.700 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:59 smithi006 ceph-mon[31842]: from='osd.2 [v2:172.21.15.6:6818/3078148920,v1:172.21.15.6:6819/3078148920]' entity='osd.2' 2024-03-10T03:46:59.700 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:46:59 smithi006 ceph-mon[31842]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:46:59.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:59 smithi071 ceph-mon[27181]: from='osd.2 [v2:172.21.15.6:6818/3078148920,v1:172.21.15.6:6819/3078148920]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-03-10T03:46:59.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:59 smithi071 ceph-mon[27181]: osdmap e17: 3 total, 2 up, 3 in 2024-03-10T03:46:59.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:59 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T03:46:59.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:59 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T03:46:59.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:59 smithi071 ceph-mon[27181]: from='osd.2 [v2:172.21.15.6:6818/3078148920,v1:172.21.15.6:6819/3078148920]' entity='osd.2' 2024-03-10T03:46:59.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:46:59 smithi071 ceph-mon[27181]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-10T03:47:00.570 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping: /dev/nvme1n1 2024-03-10T03:47:00.570 INFO:teuthology.orchestra.run.smithi006.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-10T03:47:00.570 INFO:teuthology.orchestra.run.smithi006.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme1n1 bs=1M count=10 conv=fsync 2024-03-10T03:47:00.570 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10+0 records in 2024-03-10T03:47:00.570 INFO:teuthology.orchestra.run.smithi006.stderr:10+0 records out 2024-03-10T03:47:00.570 INFO:teuthology.orchestra.run.smithi006.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0136394 s, 769 MB/s 2024-03-10T03:47:00.570 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 2024-03-10T03:47:00.570 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping successful for: 2024-03-10T03:47:00.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:00 smithi006 ceph-mon[31842]: purged_snaps scrub starts 2024-03-10T03:47:00.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:00 smithi006 ceph-mon[31842]: purged_snaps scrub ok 2024-03-10T03:47:00.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:00 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T03:47:00.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:00 smithi006 ceph-mon[31842]: osd.2 [v2:172.21.15.6:6818/3078148920,v1:172.21.15.6:6819/3078148920] boot 2024-03-10T03:47:00.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:00 smithi006 ceph-mon[31842]: osdmap e18: 3 total, 3 up, 3 in 2024-03-10T03:47:00.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:00 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T03:47:00.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:00 smithi006 ceph-mon[25256]: purged_snaps scrub starts 2024-03-10T03:47:00.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:00 smithi006 ceph-mon[25256]: purged_snaps scrub ok 2024-03-10T03:47:00.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:00 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T03:47:00.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:00 smithi006 ceph-mon[25256]: osd.2 [v2:172.21.15.6:6818/3078148920,v1:172.21.15.6:6819/3078148920] boot 2024-03-10T03:47:00.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:00 smithi006 ceph-mon[25256]: osdmap e18: 3 total, 3 up, 3 in 2024-03-10T03:47:00.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:00 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T03:47:00.840 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:00 smithi071 ceph-mon[27181]: purged_snaps scrub starts 2024-03-10T03:47:00.840 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:00 smithi071 ceph-mon[27181]: purged_snaps scrub ok 2024-03-10T03:47:00.840 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:00 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T03:47:00.840 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:00 smithi071 ceph-mon[27181]: osd.2 [v2:172.21.15.6:6818/3078148920,v1:172.21.15.6:6819/3078148920] boot 2024-03-10T03:47:00.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:00 smithi071 ceph-mon[27181]: osdmap e18: 3 total, 3 up, 3 in 2024-03-10T03:47:00.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:00 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T03:47:01.257 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph orch daemon add osd smithi006:/dev/nvme1n1 2024-03-10T03:47:01.496 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:47:01.686 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:01 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:01.686 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:01 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:01.686 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:01 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:01.686 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:01 smithi071 ceph-mon[27181]: osdmap e19: 3 total, 3 up, 3 in 2024-03-10T03:47:01.686 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:01 smithi071 ceph-mon[27181]: pgmap v83: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:01.687 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:01 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-10T03:47:01.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:01 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:01.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:01 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:01.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:01 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:01.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:01 smithi006 ceph-mon[25256]: osdmap e19: 3 total, 3 up, 3 in 2024-03-10T03:47:01.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:01 smithi006 ceph-mon[25256]: pgmap v83: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:01.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:01 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-10T03:47:01.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:01 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:01.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:01 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:01.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:01 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:01.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:01 smithi006 ceph-mon[31842]: osdmap e19: 3 total, 3 up, 3 in 2024-03-10T03:47:01.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:01 smithi006 ceph-mon[31842]: pgmap v83: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:01.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:01 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-10T03:47:03.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:02 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-10T03:47:03.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:02 smithi006 ceph-mon[25256]: osdmap e20: 3 total, 3 up, 3 in 2024-03-10T03:47:03.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:02 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-10T03:47:03.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:02 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-10T03:47:03.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:02 smithi006 ceph-mon[31842]: osdmap e20: 3 total, 3 up, 3 in 2024-03-10T03:47:03.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:02 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-10T03:47:03.192 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:02 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-10T03:47:03.192 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:02 smithi071 ceph-mon[27181]: osdmap e20: 3 total, 3 up, 3 in 2024-03-10T03:47:03.192 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:02 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-10T03:47:04.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:03 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-10T03:47:04.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:03 smithi071 ceph-mon[27181]: osdmap e21: 3 total, 3 up, 3 in 2024-03-10T03:47:04.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:03 smithi071 ceph-mon[27181]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:04.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:03 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T03:47:04.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:03 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T03:47:04.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:03 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:04.124 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 10 03:47:03 smithi006 sudo[45055]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-10T03:47:04.124 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 10 03:47:03 smithi006 sudo[45055]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-10T03:47:04.124 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 10 03:47:03 smithi006 sudo[45055]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-10T03:47:04.124 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 10 03:47:03 smithi006 sudo[45055]: pam_unix(sudo:session): session closed for user root 2024-03-10T03:47:04.124 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:03 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-10T03:47:04.125 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:03 smithi006 ceph-mon[25256]: osdmap e21: 3 total, 3 up, 3 in 2024-03-10T03:47:04.125 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:03 smithi006 ceph-mon[25256]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:04.125 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:03 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T03:47:04.125 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:03 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T03:47:04.125 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:03 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:04.125 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:03 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-10T03:47:04.125 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:03 smithi006 ceph-mon[31842]: osdmap e21: 3 total, 3 up, 3 in 2024-03-10T03:47:04.125 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:03 smithi006 ceph-mon[31842]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:04.125 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:03 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T03:47:04.125 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:03 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T03:47:04.125 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:03 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:04.401 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 10 03:47:04 smithi006 sudo[45063]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-10T03:47:04.401 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 10 03:47:04 smithi006 sudo[45063]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-10T03:47:04.401 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 10 03:47:04 smithi006 sudo[45063]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-10T03:47:04.401 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 10 03:47:04 smithi006 sudo[45063]: pam_unix(sudo:session): session closed for user root 2024-03-10T03:47:04.670 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:47:04 smithi006 sudo[45079]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme3n1 2024-03-10T03:47:04.670 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:47:04 smithi006 sudo[45079]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-10T03:47:04.670 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:47:04 smithi006 sudo[45079]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-10T03:47:04.671 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:47:04 smithi006 sudo[45079]: pam_unix(sudo:session): session closed for user root 2024-03-10T03:47:04.671 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:47:04 smithi006 sudo[45086]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme3n1 2024-03-10T03:47:04.947 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:04 smithi006 ceph-mon[25256]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:47:04.947 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:04 smithi006 ceph-mon[25256]: osdmap e22: 3 total, 3 up, 3 in 2024-03-10T03:47:04.948 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:04 smithi006 ceph-mon[25256]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-10T03:47:04.948 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:04 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:04.948 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:04 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:04.948 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:04 smithi006 ceph-mon[31842]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:47:04.948 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:04 smithi006 ceph-mon[31842]: osdmap e22: 3 total, 3 up, 3 in 2024-03-10T03:47:04.948 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:04 smithi006 ceph-mon[31842]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-10T03:47:04.948 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:04 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:04.948 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:04 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:04.948 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:47:04 smithi006 sudo[45086]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-10T03:47:04.949 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:47:04 smithi006 sudo[45086]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-10T03:47:04.949 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:47:04 smithi006 sudo[45086]: pam_unix(sudo:session): session closed for user root 2024-03-10T03:47:05.222 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 10 03:47:04 smithi006 sudo[45089]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme2n1 2024-03-10T03:47:05.222 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 10 03:47:04 smithi006 sudo[45089]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-10T03:47:05.222 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 10 03:47:04 smithi006 sudo[45089]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-10T03:47:05.222 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 10 03:47:04 smithi006 sudo[45089]: pam_unix(sudo:session): session closed for user root 2024-03-10T03:47:05.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:04 smithi071 ceph-mon[27181]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:47:05.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:04 smithi071 ceph-mon[27181]: osdmap e22: 3 total, 3 up, 3 in 2024-03-10T03:47:05.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:04 smithi071 ceph-mon[27181]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-10T03:47:05.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:04 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:05.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:04 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:05.476 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 10 03:47:05 smithi006 sudo[45113]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme2n1 2024-03-10T03:47:05.476 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 10 03:47:05 smithi006 sudo[45113]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-10T03:47:05.476 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 10 03:47:05 smithi006 sudo[45113]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-10T03:47:05.476 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 10 03:47:05 smithi006 sudo[45113]: pam_unix(sudo:session): session closed for user root 2024-03-10T03:47:05.769 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:05 smithi006 sudo[45123]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-10T03:47:05.769 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:05 smithi006 sudo[45123]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-10T03:47:05.769 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:05 smithi006 sudo[45123]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-10T03:47:05.770 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:05 smithi006 sudo[45123]: pam_unix(sudo:session): session closed for user root 2024-03-10T03:47:06.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:05 smithi006 sudo[45149]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-10T03:47:06.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:05 smithi006 sudo[45149]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-10T03:47:06.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:05 smithi006 sudo[45149]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-10T03:47:06.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:05 smithi006 sudo[45149]: pam_unix(sudo:session): session closed for user root 2024-03-10T03:47:06.289 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:06 smithi071 sudo[31064]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-10T03:47:06.289 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:06 smithi071 sudo[31064]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-10T03:47:06.289 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:06 smithi071 sudo[31064]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-10T03:47:06.290 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:06 smithi071 sudo[31064]: pam_unix(sudo:session): session closed for user root 2024-03-10T03:47:06.571 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:06 smithi006 ceph-mon[25256]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:06.571 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:06 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:06.571 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:06 smithi006 ceph-mon[25256]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-10T03:47:06.571 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:06 smithi006 ceph-mon[25256]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-10T03:47:06.571 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:06 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T03:47:06.571 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:06 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:47:06.571 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:06 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:47:06.571 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:06 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:06.571 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:06 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T03:47:06.572 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:06 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:47:06.572 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:06 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:47:06.572 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:06 smithi006 ceph-mon[31842]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:06.572 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:06 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:06.572 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:06 smithi006 ceph-mon[31842]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-10T03:47:06.572 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:06 smithi006 ceph-mon[31842]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-10T03:47:06.572 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:06 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T03:47:06.572 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:06 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:47:06.572 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:06 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:47:06.572 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:06 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:06.572 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:06 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T03:47:06.572 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:06 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:47:06.572 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:06 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:47:06.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:06 smithi071 sudo[31067]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-10T03:47:06.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:06 smithi071 sudo[31067]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-10T03:47:06.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:06 smithi071 sudo[31067]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-10T03:47:06.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:06 smithi071 sudo[31067]: pam_unix(sudo:session): session closed for user root 2024-03-10T03:47:06.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:06 smithi071 ceph-mon[27181]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:06.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:06 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:06.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:06 smithi071 ceph-mon[27181]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-10T03:47:06.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:06 smithi071 ceph-mon[27181]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-10T03:47:06.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:06 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T03:47:06.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:06 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:47:06.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:06 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:47:06.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:06 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:06.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:06 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T03:47:06.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:06 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:47:06.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:06 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:47:06.836 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:06 smithi006 sudo[45196]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-10T03:47:06.836 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:06 smithi006 sudo[45196]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-10T03:47:06.836 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:06 smithi006 sudo[45196]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-10T03:47:06.836 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:06 smithi006 sudo[45196]: pam_unix(sudo:session): session closed for user root 2024-03-10T03:47:07.299 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:06 smithi006 sudo[45206]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-10T03:47:07.299 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:06 smithi006 sudo[45206]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-10T03:47:07.299 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:06 smithi006 sudo[45206]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-10T03:47:07.299 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:06 smithi006 sudo[45206]: pam_unix(sudo:session): session closed for user root 2024-03-10T03:47:07.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[25256]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-10T03:47:07.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/3393611407' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ecb7b3ca-493f-4837-9772-146a4f323ba4"}]: dispatch 2024-03-10T03:47:07.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[25256]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-10T03:47:07.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/3393611407' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ecb7b3ca-493f-4837-9772-146a4f323ba4"}]': finished 2024-03-10T03:47:07.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[25256]: osdmap e23: 4 total, 3 up, 4 in 2024-03-10T03:47:07.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T03:47:07.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:07.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[25256]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-10T03:47:07.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T03:47:07.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:47:07.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:47:07.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[25256]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-10T03:47:07.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[25256]: pgmap v90: 1 pgs: 1 creating+peering; 0 B data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:07.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[31842]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-10T03:47:07.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/3393611407' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ecb7b3ca-493f-4837-9772-146a4f323ba4"}]: dispatch 2024-03-10T03:47:07.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[31842]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-10T03:47:07.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/3393611407' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ecb7b3ca-493f-4837-9772-146a4f323ba4"}]': finished 2024-03-10T03:47:07.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[31842]: osdmap e23: 4 total, 3 up, 4 in 2024-03-10T03:47:07.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T03:47:07.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:07.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[31842]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-10T03:47:07.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T03:47:07.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:47:07.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:47:07.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[31842]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-10T03:47:07.589 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:07 smithi006 ceph-mon[31842]: pgmap v90: 1 pgs: 1 creating+peering; 0 B data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:07.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:07 smithi071 ceph-mon[27181]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-10T03:47:07.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:07 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/3393611407' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ecb7b3ca-493f-4837-9772-146a4f323ba4"}]: dispatch 2024-03-10T03:47:07.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:07 smithi071 ceph-mon[27181]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-10T03:47:07.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:07 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/3393611407' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ecb7b3ca-493f-4837-9772-146a4f323ba4"}]': finished 2024-03-10T03:47:07.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:07 smithi071 ceph-mon[27181]: osdmap e23: 4 total, 3 up, 4 in 2024-03-10T03:47:07.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:07 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T03:47:07.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:07 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:07.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:07 smithi071 ceph-mon[27181]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-10T03:47:07.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:07 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T03:47:07.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:07 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:47:07.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:07 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:47:07.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:07 smithi071 ceph-mon[27181]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-10T03:47:07.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:07 smithi071 ceph-mon[27181]: pgmap v90: 1 pgs: 1 creating+peering; 0 B data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:08.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:08 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/3161169347' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T03:47:08.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:08 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/3161169347' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T03:47:08.590 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:08 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/3161169347' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T03:47:09.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:09 smithi006 ceph-mon[25256]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:09.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:09 smithi006 ceph-mon[31842]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:09.590 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:09 smithi071 ceph-mon[27181]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:11.293 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:11 smithi006 ceph-mon[25256]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:11.293 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:11 smithi006 ceph-mon[31842]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:11.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:11 smithi071 ceph-mon[27181]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:13.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:13 smithi006 ceph-mon[25256]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:13.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:13 smithi006 ceph-mon[31842]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:13.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:13 smithi071 ceph-mon[27181]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:14.481 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:14 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:14.481 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:14 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:14.481 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:14 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:14.482 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:14 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-10T03:47:14.482 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:14 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:14.482 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:14 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:14.482 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:14 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:14.482 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:14 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:14.482 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:14 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-10T03:47:14.482 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:14 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:14.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:14 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:14.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:14 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:14.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:14 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:14.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:14 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-10T03:47:14.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:14 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:15.803 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:15 smithi006 ceph-mon[31842]: Deploying daemon osd.3 on smithi006 2024-03-10T03:47:15.804 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:15 smithi006 ceph-mon[31842]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:15.804 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:15 smithi006 ceph-mon[25256]: Deploying daemon osd.3 on smithi006 2024-03-10T03:47:15.804 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:15 smithi006 ceph-mon[25256]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:15.840 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:15 smithi071 ceph-mon[27181]: Deploying daemon osd.3 on smithi006 2024-03-10T03:47:15.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:15 smithi071 ceph-mon[27181]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:17.235 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:17 smithi006 ceph-mon[25256]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:17.235 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:17 smithi006 ceph-mon[31842]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:17.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:17 smithi071 ceph-mon[27181]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:19.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:19 smithi006 ceph-mon[25256]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:19.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:19 smithi006 ceph-mon[31842]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:19.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:19 smithi071 ceph-mon[27181]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:22.001 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:22.001 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:22.001 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:22.001 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:21 smithi071 ceph-mon[27181]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:22.001 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:22.001 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:22.002 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:47:22.002 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:22.002 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:47:22.002 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:22.071 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:22.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:22.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:22.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:21 smithi006 ceph-mon[25256]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:22.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:22.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:22.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:47:22.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:22.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:47:22.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:22.072 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:22.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:22.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:22.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:21 smithi006 ceph-mon[31842]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:22.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:22.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:22.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:47:22.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:22.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:47:22.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:23.061 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:23 smithi006 ceph-mon[25256]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:23.061 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:23 smithi006 ceph-mon[31842]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:23.308 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:23 smithi071 ceph-mon[27181]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:24.757 INFO:teuthology.orchestra.run.smithi006.stdout:Created osd(s) 3 on host 'smithi006' 2024-03-10T03:47:24.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:24 smithi006 ceph-mon[25256]: from='osd.3 [v2:172.21.15.6:6826/388589465,v1:172.21.15.6:6827/388589465]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-10T03:47:24.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:24 smithi006 ceph-mon[25256]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-10T03:47:24.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:24 smithi006 ceph-mon[31842]: from='osd.3 [v2:172.21.15.6:6826/388589465,v1:172.21.15.6:6827/388589465]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-10T03:47:24.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:24 smithi006 ceph-mon[31842]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-10T03:47:25.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:24 smithi071 ceph-mon[27181]: from='osd.3 [v2:172.21.15.6:6826/388589465,v1:172.21.15.6:6827/388589465]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-10T03:47:25.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:24 smithi071 ceph-mon[27181]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-10T03:47:25.439 DEBUG:teuthology.orchestra.run.smithi006:osd.3> sudo journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.3.service 2024-03-10T03:47:25.441 INFO:tasks.cephadm:Deploying osd.4 on smithi071 with /dev/nvme4n1... 2024-03-10T03:47:25.441 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-10T03:47:25.686 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T03:47:25.808 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:25 smithi071 ceph-mon[27181]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-10T03:47:25.808 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:25 smithi071 ceph-mon[27181]: from='osd.3 [v2:172.21.15.6:6826/388589465,v1:172.21.15.6:6827/388589465]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-10T03:47:25.808 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:25 smithi071 ceph-mon[27181]: osdmap e24: 4 total, 3 up, 4 in 2024-03-10T03:47:25.808 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:25 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T03:47:25.808 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:25 smithi071 ceph-mon[27181]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-10T03:47:25.808 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:25 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:25.808 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:25 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:25.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:25 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:47:25.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:25 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:47:25.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:25 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:47:25.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:25 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:47:25.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:25 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:47:25.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:25 smithi071 ceph-mon[27181]: Adjusting osd_memory_target on smithi006 to 3761M 2024-03-10T03:47:25.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:25 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:25.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:25 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:25.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:25 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:47:25.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:25 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:25.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:25 smithi071 ceph-mon[27181]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:25.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:25 smithi071 ceph-mon[27181]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-03-10T03:47:25.809 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:25 smithi071 ceph-mon[27181]: osdmap e25: 4 total, 3 up, 4 in 2024-03-10T03:47:25.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:25 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T03:47:25.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[31842]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-10T03:47:25.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[31842]: from='osd.3 [v2:172.21.15.6:6826/388589465,v1:172.21.15.6:6827/388589465]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-10T03:47:25.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[31842]: osdmap e24: 4 total, 3 up, 4 in 2024-03-10T03:47:25.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T03:47:25.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[31842]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-10T03:47:25.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:25.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:25.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:47:25.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:47:25.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:47:25.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:47:25.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:47:25.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[31842]: Adjusting osd_memory_target on smithi006 to 3761M 2024-03-10T03:47:25.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:25.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:25.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:47:25.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:25.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[31842]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:25.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[31842]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-03-10T03:47:25.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[31842]: osdmap e25: 4 total, 3 up, 4 in 2024-03-10T03:47:25.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T03:47:25.839 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-3[47139]: 2024-03-10T03:47:25.610+0000 7fb692e8d700 -1 osd.3 0 waiting for initial osdmap 2024-03-10T03:47:25.839 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-3[47139]: 2024-03-10T03:47:25.614+0000 7fb68ec80700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-10T03:47:25.840 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[25256]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-10T03:47:25.840 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[25256]: from='osd.3 [v2:172.21.15.6:6826/388589465,v1:172.21.15.6:6827/388589465]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-10T03:47:25.840 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[25256]: osdmap e24: 4 total, 3 up, 4 in 2024-03-10T03:47:25.840 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T03:47:25.840 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[25256]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-10T03:47:25.841 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:25.841 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:25.841 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:47:25.841 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:47:25.841 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:47:25.842 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:47:25.842 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:47:25.842 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[25256]: Adjusting osd_memory_target on smithi006 to 3761M 2024-03-10T03:47:25.842 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:25.842 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:25.842 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:47:25.843 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:25.843 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[25256]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:25.843 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[25256]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-03-10T03:47:25.843 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[25256]: osdmap e25: 4 total, 3 up, 4 in 2024-03-10T03:47:25.843 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:25 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T03:47:26.924 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:26 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T03:47:26.924 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:26 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T03:47:27.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:26 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T03:47:27.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:26 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T03:47:27.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:26 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T03:47:27.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:26 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T03:47:27.449 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping: /dev/nvme4n1 2024-03-10T03:47:27.449 INFO:teuthology.orchestra.run.smithi071.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-10T03:47:27.449 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-10T03:47:27.449 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10+0 records in 2024-03-10T03:47:27.450 INFO:teuthology.orchestra.run.smithi071.stderr:10+0 records out 2024-03-10T03:47:27.450 INFO:teuthology.orchestra.run.smithi071.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0157262 s, 667 MB/s 2024-03-10T03:47:27.450 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping successful for: 2024-03-10T03:47:27.705 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:27 smithi071 ceph-mon[27181]: purged_snaps scrub starts 2024-03-10T03:47:27.706 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:27 smithi071 ceph-mon[27181]: purged_snaps scrub ok 2024-03-10T03:47:27.706 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:27 smithi071 ceph-mon[27181]: osd.3 [v2:172.21.15.6:6826/388589465,v1:172.21.15.6:6827/388589465] boot 2024-03-10T03:47:27.706 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:27 smithi071 ceph-mon[27181]: osdmap e26: 4 total, 4 up, 4 in 2024-03-10T03:47:27.706 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:27 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T03:47:27.706 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:27 smithi071 ceph-mon[27181]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:27.973 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph orch daemon add osd smithi071:/dev/nvme4n1 2024-03-10T03:47:28.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:27 smithi006 ceph-mon[25256]: purged_snaps scrub starts 2024-03-10T03:47:28.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:27 smithi006 ceph-mon[25256]: purged_snaps scrub ok 2024-03-10T03:47:28.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:27 smithi006 ceph-mon[25256]: osd.3 [v2:172.21.15.6:6826/388589465,v1:172.21.15.6:6827/388589465] boot 2024-03-10T03:47:28.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:27 smithi006 ceph-mon[25256]: osdmap e26: 4 total, 4 up, 4 in 2024-03-10T03:47:28.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:27 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T03:47:28.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:27 smithi006 ceph-mon[25256]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:28.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:27 smithi006 ceph-mon[31842]: purged_snaps scrub starts 2024-03-10T03:47:28.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:27 smithi006 ceph-mon[31842]: purged_snaps scrub ok 2024-03-10T03:47:28.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:27 smithi006 ceph-mon[31842]: osd.3 [v2:172.21.15.6:6826/388589465,v1:172.21.15.6:6827/388589465] boot 2024-03-10T03:47:28.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:27 smithi006 ceph-mon[31842]: osdmap e26: 4 total, 4 up, 4 in 2024-03-10T03:47:28.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:27 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T03:47:28.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:27 smithi006 ceph-mon[31842]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 87 MiB used, 268 GiB / 268 GiB avail 2024-03-10T03:47:28.205 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T03:47:28.961 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:28 smithi071 ceph-mon[27181]: osdmap e27: 4 total, 4 up, 4 in 2024-03-10T03:47:29.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:28 smithi006 ceph-mon[25256]: osdmap e27: 4 total, 4 up, 4 in 2024-03-10T03:47:29.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:28 smithi006 ceph-mon[31842]: osdmap e27: 4 total, 4 up, 4 in 2024-03-10T03:47:30.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:29 smithi006 ceph-mon[25256]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:30.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:29 smithi006 ceph-mon[31842]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:30.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:29 smithi071 ceph-mon[27181]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:30.894 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:30 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T03:47:30.895 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:30 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T03:47:30.895 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:30 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:30.895 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:30 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T03:47:30.895 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:30 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T03:47:30.895 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:30 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:30.900 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:30 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T03:47:30.900 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:30 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T03:47:30.900 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:30 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:31.948 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:31 smithi071 ceph-mon[27181]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:47:31.948 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:31 smithi071 ceph-mon[27181]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:32.032 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:31 smithi006 ceph-mon[25256]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:47:32.032 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:31 smithi006 ceph-mon[25256]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:32.032 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:31 smithi006 ceph-mon[31842]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:47:32.033 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:31 smithi006 ceph-mon[31842]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:33.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:32 smithi006 ceph-mon[25256]: from='client.? 172.21.15.71:0/917597352' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cdd2e4a-82c4-422a-a716-1ac8527991a5"}]: dispatch 2024-03-10T03:47:33.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:32 smithi006 ceph-mon[25256]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cdd2e4a-82c4-422a-a716-1ac8527991a5"}]: dispatch 2024-03-10T03:47:33.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:32 smithi006 ceph-mon[25256]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5cdd2e4a-82c4-422a-a716-1ac8527991a5"}]': finished 2024-03-10T03:47:33.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:32 smithi006 ceph-mon[25256]: osdmap e28: 5 total, 4 up, 5 in 2024-03-10T03:47:33.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:32 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T03:47:33.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:32 smithi006 ceph-mon[31842]: from='client.? 172.21.15.71:0/917597352' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cdd2e4a-82c4-422a-a716-1ac8527991a5"}]: dispatch 2024-03-10T03:47:33.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:32 smithi006 ceph-mon[31842]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cdd2e4a-82c4-422a-a716-1ac8527991a5"}]: dispatch 2024-03-10T03:47:33.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:32 smithi006 ceph-mon[31842]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5cdd2e4a-82c4-422a-a716-1ac8527991a5"}]': finished 2024-03-10T03:47:33.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:32 smithi006 ceph-mon[31842]: osdmap e28: 5 total, 4 up, 5 in 2024-03-10T03:47:33.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:32 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T03:47:33.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:32 smithi071 ceph-mon[27181]: from='client.? 172.21.15.71:0/917597352' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cdd2e4a-82c4-422a-a716-1ac8527991a5"}]: dispatch 2024-03-10T03:47:33.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:32 smithi071 ceph-mon[27181]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5cdd2e4a-82c4-422a-a716-1ac8527991a5"}]: dispatch 2024-03-10T03:47:33.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:32 smithi071 ceph-mon[27181]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5cdd2e4a-82c4-422a-a716-1ac8527991a5"}]': finished 2024-03-10T03:47:33.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:32 smithi071 ceph-mon[27181]: osdmap e28: 5 total, 4 up, 5 in 2024-03-10T03:47:33.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:32 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T03:47:33.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:33 smithi006 ceph-mon[25256]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:33.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:33 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:33.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:33 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:33.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:33 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:33.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:33 smithi006 ceph-mon[31842]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:33.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:33 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:33.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:33 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:33.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:33 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:34.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:33 smithi071 ceph-mon[27181]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:34.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:33 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:34.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:33 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:34.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:33 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:35.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:34 smithi006 ceph-mon[25256]: Detected new or changed devices on smithi006 2024-03-10T03:47:35.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:34 smithi006 ceph-mon[25256]: from='client.? 172.21.15.71:0/3024927127' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T03:47:35.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:34 smithi006 ceph-mon[31842]: Detected new or changed devices on smithi006 2024-03-10T03:47:35.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:34 smithi006 ceph-mon[31842]: from='client.? 172.21.15.71:0/3024927127' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T03:47:35.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:34 smithi071 ceph-mon[27181]: Detected new or changed devices on smithi006 2024-03-10T03:47:35.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:34 smithi071 ceph-mon[27181]: from='client.? 172.21.15.71:0/3024927127' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T03:47:36.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:35 smithi006 ceph-mon[25256]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:36.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:35 smithi006 ceph-mon[31842]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:36.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:35 smithi071 ceph-mon[27181]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:37.284 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:37 smithi071 ceph-mon[27181]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:37.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:37 smithi006 ceph-mon[25256]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:37.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:37 smithi006 ceph-mon[31842]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:39.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:39 smithi006 ceph-mon[25256]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:39.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:39 smithi006 ceph-mon[31842]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:39.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:39 smithi071 ceph-mon[27181]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:40.327 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:40 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-10T03:47:40.327 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:40 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:40.327 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:40 smithi071 ceph-mon[27181]: Deploying daemon osd.4 on smithi071 2024-03-10T03:47:40.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:40 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-10T03:47:40.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:40 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:40.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:40 smithi006 ceph-mon[25256]: Deploying daemon osd.4 on smithi071 2024-03-10T03:47:40.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:40 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-10T03:47:40.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:40 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:40.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:40 smithi006 ceph-mon[31842]: Deploying daemon osd.4 on smithi071 2024-03-10T03:47:42.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:41 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:42.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:41 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:42.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:41 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:42.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:41 smithi006 ceph-mon[25256]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:42.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:41 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:42.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:41 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:42.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:41 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:42.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:41 smithi006 ceph-mon[31842]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:42.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:41 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:42.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:41 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:42.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:41 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:42.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:41 smithi071 ceph-mon[27181]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:43.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:43 smithi071 ceph-mon[27181]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:43.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:43 smithi006 ceph-mon[25256]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:43.336 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:43 smithi006 ceph-mon[31842]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:45.193 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:45 smithi071 ceph-mon[27181]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:45.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:45 smithi006 ceph-mon[25256]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:45.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:45 smithi006 ceph-mon[31842]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:46.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:46 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:46.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:46 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:47:46.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:46 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:46.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:46 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:47:46.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:46 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:46.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:46 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:47.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:46 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:47.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:46 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:47:47.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:46 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:47.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:46 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:47:47.088 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:46 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:47.088 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:46 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:47.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:46 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:47.089 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:46 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:47:47.089 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:46 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:47.089 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:46 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:47:47.089 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:46 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:47.089 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:46 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:48.056 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:47 smithi071 ceph-mon[27181]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:48.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:47 smithi006 ceph-mon[25256]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:48.086 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:47 smithi006 ceph-mon[31842]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:49.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:48 smithi071 ceph-mon[27181]: from='osd.4 [v2:172.21.15.71:6800/150275056,v1:172.21.15.71:6801/150275056]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-10T03:47:49.061 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:48 smithi071 ceph-mon[27181]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-10T03:47:49.304 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:48 smithi006 ceph-mon[25256]: from='osd.4 [v2:172.21.15.71:6800/150275056,v1:172.21.15.71:6801/150275056]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-10T03:47:49.304 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:48 smithi006 ceph-mon[25256]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-10T03:47:49.305 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:48 smithi006 ceph-mon[31842]: from='osd.4 [v2:172.21.15.71:6800/150275056,v1:172.21.15.71:6801/150275056]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-10T03:47:49.305 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:48 smithi006 ceph-mon[31842]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-10T03:47:49.340 INFO:teuthology.orchestra.run.smithi071.stdout:Created osd(s) 4 on host 'smithi071' 2024-03-10T03:47:49.991 DEBUG:teuthology.orchestra.run.smithi071:osd.4> sudo journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.4.service 2024-03-10T03:47:49.993 INFO:tasks.cephadm:Deploying osd.5 on smithi071 with /dev/nvme3n1... 2024-03-10T03:47:49.993 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph-volume lvm zap /dev/nvme3n1 2024-03-10T03:47:50.231 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[25256]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:50.231 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[25256]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-10T03:47:50.232 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[25256]: from='osd.4 [v2:172.21.15.71:6800/150275056,v1:172.21.15.71:6801/150275056]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-10T03:47:50.232 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[25256]: osdmap e29: 5 total, 4 up, 5 in 2024-03-10T03:47:50.232 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T03:47:50.232 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[25256]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-10T03:47:50.232 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:50.232 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:50.232 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:47:50.232 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:47:50.232 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:50.232 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:50.232 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:47:50.232 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:50.232 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[25256]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-03-10T03:47:50.233 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[25256]: osdmap e30: 5 total, 4 up, 5 in 2024-03-10T03:47:50.233 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T03:47:50.233 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[31842]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:50.233 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[31842]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-10T03:47:50.233 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[31842]: from='osd.4 [v2:172.21.15.71:6800/150275056,v1:172.21.15.71:6801/150275056]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-10T03:47:50.233 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[31842]: osdmap e29: 5 total, 4 up, 5 in 2024-03-10T03:47:50.233 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T03:47:50.233 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[31842]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-10T03:47:50.233 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:50.233 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:50.233 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:47:50.233 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:47:50.233 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:50.234 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:50.234 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:47:50.234 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:50.234 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[31842]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-03-10T03:47:50.234 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[31842]: osdmap e30: 5 total, 4 up, 5 in 2024-03-10T03:47:50.234 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:49 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T03:47:50.277 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T03:47:50.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:49 smithi071 ceph-mon[27181]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:50.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:49 smithi071 ceph-mon[27181]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-10T03:47:50.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:49 smithi071 ceph-mon[27181]: from='osd.4 [v2:172.21.15.71:6800/150275056,v1:172.21.15.71:6801/150275056]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-10T03:47:50.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:49 smithi071 ceph-mon[27181]: osdmap e29: 5 total, 4 up, 5 in 2024-03-10T03:47:50.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:49 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T03:47:50.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:49 smithi071 ceph-mon[27181]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-10T03:47:50.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:49 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:50.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:49 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:50.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:49 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:47:50.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:49 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:47:50.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:49 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:50.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:49 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:50.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:49 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:47:50.342 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:49 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:50.342 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:49 smithi071 ceph-mon[27181]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-03-10T03:47:50.342 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:49 smithi071 ceph-mon[27181]: osdmap e30: 5 total, 4 up, 5 in 2024-03-10T03:47:50.342 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:49 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T03:47:50.710 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:47:50 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:47:50.417+0000 7fc3136fc700 -1 osd.4 0 waiting for initial osdmap 2024-03-10T03:47:50.710 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:47:50 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:47:50.423+0000 7fc30b4e7700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-10T03:47:51.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:50 smithi071 ceph-mon[27181]: Adjusting osd_memory_target on smithi071 to 16064M 2024-03-10T03:47:51.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:50 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T03:47:51.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:50 smithi071 ceph-mon[27181]: from='osd.4 ' entity='osd.4' 2024-03-10T03:47:51.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:50 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:51.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:50 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:51.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:50 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:51.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:50 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T03:47:51.336 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:50 smithi006 ceph-mon[31842]: Adjusting osd_memory_target on smithi071 to 16064M 2024-03-10T03:47:51.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:50 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T03:47:51.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:50 smithi006 ceph-mon[31842]: from='osd.4 ' entity='osd.4' 2024-03-10T03:47:51.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:50 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:51.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:50 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:51.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:50 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:51.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:50 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T03:47:51.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:50 smithi006 ceph-mon[25256]: Adjusting osd_memory_target on smithi071 to 16064M 2024-03-10T03:47:51.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:50 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T03:47:51.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:50 smithi006 ceph-mon[25256]: from='osd.4 ' entity='osd.4' 2024-03-10T03:47:51.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:50 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:51.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:50 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:51.338 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:50 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:47:51.338 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:50 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T03:47:52.063 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:51 smithi006 ceph-mon[25256]: purged_snaps scrub starts 2024-03-10T03:47:52.063 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:51 smithi006 ceph-mon[25256]: purged_snaps scrub ok 2024-03-10T03:47:52.063 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:51 smithi006 ceph-mon[25256]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:52.063 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:51 smithi006 ceph-mon[25256]: osd.4 [v2:172.21.15.71:6800/150275056,v1:172.21.15.71:6801/150275056] boot 2024-03-10T03:47:52.063 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:51 smithi006 ceph-mon[25256]: osdmap e31: 5 total, 5 up, 5 in 2024-03-10T03:47:52.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:51 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T03:47:52.064 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:51 smithi006 ceph-mon[31842]: purged_snaps scrub starts 2024-03-10T03:47:52.064 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:51 smithi006 ceph-mon[31842]: purged_snaps scrub ok 2024-03-10T03:47:52.064 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:51 smithi006 ceph-mon[31842]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:52.064 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:51 smithi006 ceph-mon[31842]: osd.4 [v2:172.21.15.71:6800/150275056,v1:172.21.15.71:6801/150275056] boot 2024-03-10T03:47:52.064 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:51 smithi006 ceph-mon[31842]: osdmap e31: 5 total, 5 up, 5 in 2024-03-10T03:47:52.064 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:51 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T03:47:52.155 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping: /dev/nvme3n1 2024-03-10T03:47:52.155 INFO:teuthology.orchestra.run.smithi071.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-10T03:47:52.155 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme3n1 bs=1M count=10 conv=fsync 2024-03-10T03:47:52.155 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10+0 records in 2024-03-10T03:47:52.155 INFO:teuthology.orchestra.run.smithi071.stderr:10+0 records out 2024-03-10T03:47:52.155 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129505 s, 810 MB/s 2024-03-10T03:47:52.155 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping successful for: 2024-03-10T03:47:52.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:51 smithi071 ceph-mon[27181]: purged_snaps scrub starts 2024-03-10T03:47:52.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:51 smithi071 ceph-mon[27181]: purged_snaps scrub ok 2024-03-10T03:47:52.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:51 smithi071 ceph-mon[27181]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-10T03:47:52.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:51 smithi071 ceph-mon[27181]: osd.4 [v2:172.21.15.71:6800/150275056,v1:172.21.15.71:6801/150275056] boot 2024-03-10T03:47:52.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:51 smithi071 ceph-mon[27181]: osdmap e31: 5 total, 5 up, 5 in 2024-03-10T03:47:52.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:51 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T03:47:52.731 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph orch daemon add osd smithi071:/dev/nvme3n1 2024-03-10T03:47:52.968 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T03:47:53.238 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:52 smithi071 ceph-mon[27181]: osdmap e32: 5 total, 5 up, 5 in 2024-03-10T03:47:53.238 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:52 smithi071 ceph-mon[27181]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-10T03:47:53.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:52 smithi006 ceph-mon[25256]: osdmap e32: 5 total, 5 up, 5 in 2024-03-10T03:47:53.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:52 smithi006 ceph-mon[25256]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-10T03:47:53.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:52 smithi006 ceph-mon[31842]: osdmap e32: 5 total, 5 up, 5 in 2024-03-10T03:47:53.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:52 smithi006 ceph-mon[31842]: pgmap v122: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-10T03:47:54.128 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:53 smithi071 ceph-mon[27181]: osdmap e33: 5 total, 5 up, 5 in 2024-03-10T03:47:54.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:53 smithi006 ceph-mon[25256]: osdmap e33: 5 total, 5 up, 5 in 2024-03-10T03:47:54.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:53 smithi006 ceph-mon[31842]: osdmap e33: 5 total, 5 up, 5 in 2024-03-10T03:47:55.329 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:54 smithi071 ceph-mon[27181]: osdmap e34: 5 total, 5 up, 5 in 2024-03-10T03:47:55.329 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:54 smithi071 ceph-mon[27181]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:47:55.329 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:54 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T03:47:55.329 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:54 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T03:47:55.329 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:54 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:55.329 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:54 smithi071 ceph-mon[27181]: pgmap v125: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-10T03:47:55.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:54 smithi006 ceph-mon[25256]: osdmap e34: 5 total, 5 up, 5 in 2024-03-10T03:47:55.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:54 smithi006 ceph-mon[25256]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:47:55.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:54 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T03:47:55.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:54 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T03:47:55.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:54 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:55.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:54 smithi006 ceph-mon[25256]: pgmap v125: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-10T03:47:55.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:54 smithi006 ceph-mon[31842]: osdmap e34: 5 total, 5 up, 5 in 2024-03-10T03:47:55.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:54 smithi006 ceph-mon[31842]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:/dev/nvme3n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:47:55.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:54 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T03:47:55.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:54 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T03:47:55.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:54 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:47:55.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:54 smithi006 ceph-mon[31842]: pgmap v125: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-10T03:47:56.023 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:55 smithi071 ceph-mon[27181]: osdmap e35: 5 total, 5 up, 5 in 2024-03-10T03:47:56.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:55 smithi006 ceph-mon[25256]: osdmap e35: 5 total, 5 up, 5 in 2024-03-10T03:47:56.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:55 smithi006 ceph-mon[31842]: osdmap e35: 5 total, 5 up, 5 in 2024-03-10T03:47:57.336 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:57 smithi006 ceph-mon[31842]: pgmap v127: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-10T03:47:57.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:57 smithi006 ceph-mon[25256]: pgmap v127: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-10T03:47:57.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:57 smithi071 ceph-mon[27181]: pgmap v127: 1 pgs: 1 remapped+peering; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-10T03:47:58.336 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:58 smithi006 ceph-mon[31842]: from='client.? 172.21.15.71:0/3917330934' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1f68af1-e235-4363-8e31-19ea142661fb"}]: dispatch 2024-03-10T03:47:58.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:58 smithi006 ceph-mon[31842]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1f68af1-e235-4363-8e31-19ea142661fb"}]: dispatch 2024-03-10T03:47:58.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:58 smithi006 ceph-mon[31842]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1f68af1-e235-4363-8e31-19ea142661fb"}]': finished 2024-03-10T03:47:58.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:58 smithi006 ceph-mon[31842]: osdmap e36: 6 total, 5 up, 6 in 2024-03-10T03:47:58.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:58 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T03:47:58.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:58 smithi006 ceph-mon[25256]: from='client.? 172.21.15.71:0/3917330934' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1f68af1-e235-4363-8e31-19ea142661fb"}]: dispatch 2024-03-10T03:47:58.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:58 smithi006 ceph-mon[25256]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1f68af1-e235-4363-8e31-19ea142661fb"}]: dispatch 2024-03-10T03:47:58.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:58 smithi006 ceph-mon[25256]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1f68af1-e235-4363-8e31-19ea142661fb"}]': finished 2024-03-10T03:47:58.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:58 smithi006 ceph-mon[25256]: osdmap e36: 6 total, 5 up, 6 in 2024-03-10T03:47:58.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:58 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T03:47:58.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:58 smithi071 ceph-mon[27181]: from='client.? 172.21.15.71:0/3917330934' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1f68af1-e235-4363-8e31-19ea142661fb"}]: dispatch 2024-03-10T03:47:58.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:58 smithi071 ceph-mon[27181]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1f68af1-e235-4363-8e31-19ea142661fb"}]: dispatch 2024-03-10T03:47:58.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:58 smithi071 ceph-mon[27181]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1f68af1-e235-4363-8e31-19ea142661fb"}]': finished 2024-03-10T03:47:58.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:58 smithi071 ceph-mon[27181]: osdmap e36: 6 total, 5 up, 6 in 2024-03-10T03:47:58.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:58 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T03:47:59.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:59 smithi071 ceph-mon[27181]: from='client.? 172.21.15.71:0/785053100' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T03:47:59.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:47:59 smithi071 ceph-mon[27181]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-10T03:47:59.336 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:59 smithi006 ceph-mon[31842]: from='client.? 172.21.15.71:0/785053100' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T03:47:59.336 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:47:59 smithi006 ceph-mon[31842]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-10T03:47:59.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:59 smithi006 ceph-mon[25256]: from='client.? 172.21.15.71:0/785053100' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T03:47:59.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:47:59 smithi006 ceph-mon[25256]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-10T03:48:01.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:00 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:01.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:00 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:01.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:00 smithi071 ceph-mon[27181]: Detected new or changed devices on smithi071 2024-03-10T03:48:01.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:00 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:01.336 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:00 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:01.336 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:00 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:01.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:00 smithi006 ceph-mon[31842]: Detected new or changed devices on smithi071 2024-03-10T03:48:01.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:00 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:01.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:00 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:01.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:00 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:01.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:00 smithi006 ceph-mon[25256]: Detected new or changed devices on smithi071 2024-03-10T03:48:01.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:00 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:02.206 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:01 smithi071 ceph-mon[27181]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-10T03:48:02.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:01 smithi006 ceph-mon[25256]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-10T03:48:02.336 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:01 smithi006 ceph-mon[31842]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-10T03:48:04.264 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:03 smithi071 ceph-mon[27181]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-10T03:48:04.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:03 smithi006 ceph-mon[25256]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-10T03:48:04.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:03 smithi006 ceph-mon[31842]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-10T03:48:05.322 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:04 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-10T03:48:05.322 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:04 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:05.322 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:04 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-10T03:48:05.322 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:04 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:05.329 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:04 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-10T03:48:05.329 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:04 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:06.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:05 smithi071 ceph-mon[27181]: Deploying daemon osd.5 on smithi071 2024-03-10T03:48:06.312 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:05 smithi071 ceph-mon[27181]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-10T03:48:06.336 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:05 smithi006 ceph-mon[31842]: Deploying daemon osd.5 on smithi071 2024-03-10T03:48:06.336 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:05 smithi006 ceph-mon[31842]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-10T03:48:06.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:05 smithi006 ceph-mon[25256]: Deploying daemon osd.5 on smithi071 2024-03-10T03:48:06.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:05 smithi006 ceph-mon[25256]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-10T03:48:07.081 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:07 smithi071 ceph-mon[27181]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-10T03:48:07.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:07 smithi006 ceph-mon[25256]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-10T03:48:07.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:07 smithi006 ceph-mon[31842]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-10T03:48:09.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:09 smithi006 ceph-mon[25256]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-10T03:48:09.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:09 smithi006 ceph-mon[31842]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-10T03:48:09.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:09 smithi071 ceph-mon[27181]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-03-10T03:48:10.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:10 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:48:10.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:10 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:10.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:10 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:10.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:10 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:48:10.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:10 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:10.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:10 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:10.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:10 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:48:10.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:10 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:10.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:10 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:10.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:10 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:48:10.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:10 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:10.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:10 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:10.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:10 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:48:10.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:10 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:10.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:10 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:10.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:10 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:48:10.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:10 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:10.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:10 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:11.767 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:11 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:11.767 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:11 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:11.767 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:11 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:11.768 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:11 smithi071 ceph-mon[27181]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-10T03:48:11.810 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:11 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:11.810 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:11 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:11.810 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:11 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:11.810 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:11 smithi006 ceph-mon[25256]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-10T03:48:11.810 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:11 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:11.810 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:11 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:11.811 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:11 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:11.811 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:11 smithi006 ceph-mon[31842]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-10T03:48:13.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:13 smithi071 ceph-mon[27181]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-10T03:48:13.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:13 smithi006 ceph-mon[25256]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-10T03:48:13.336 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:13 smithi006 ceph-mon[31842]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-10T03:48:14.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:14 smithi071 ceph-mon[27181]: from='osd.5 [v2:172.21.15.71:6808/3012063760,v1:172.21.15.71:6809/3012063760]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-10T03:48:14.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:14 smithi071 ceph-mon[27181]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-10T03:48:14.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:14 smithi006 ceph-mon[25256]: from='osd.5 [v2:172.21.15.71:6808/3012063760,v1:172.21.15.71:6809/3012063760]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-10T03:48:14.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:14 smithi006 ceph-mon[25256]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-10T03:48:14.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:14 smithi006 ceph-mon[31842]: from='osd.5 [v2:172.21.15.71:6808/3012063760,v1:172.21.15.71:6809/3012063760]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-10T03:48:14.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:14 smithi006 ceph-mon[31842]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-10T03:48:14.478 INFO:teuthology.orchestra.run.smithi071.stdout:Created osd(s) 5 on host 'smithi071' 2024-03-10T03:48:15.051 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:15 smithi071 ceph-mon[27181]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-10T03:48:15.051 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:15 smithi071 ceph-mon[27181]: osdmap e37: 6 total, 5 up, 6 in 2024-03-10T03:48:15.051 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:15 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T03:48:15.051 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:15 smithi071 ceph-mon[27181]: from='osd.5 [v2:172.21.15.71:6808/3012063760,v1:172.21.15.71:6809/3012063760]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-10T03:48:15.051 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:15 smithi071 ceph-mon[27181]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-10T03:48:15.051 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:15 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:15.051 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:15 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:15.052 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:15 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:48:15.052 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:15 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:48:15.052 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:15 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:48:15.052 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:15 smithi071 ceph-mon[27181]: Adjusting osd_memory_target on smithi071 to 8032M 2024-03-10T03:48:15.052 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:15 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:15.052 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:15 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:15.052 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:15 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:48:15.052 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:15 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:15.052 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:15 smithi071 ceph-mon[27181]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-10T03:48:15.078 DEBUG:teuthology.orchestra.run.smithi071:osd.5> sudo journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.5.service 2024-03-10T03:48:15.081 INFO:tasks.cephadm:Deploying osd.6 on smithi071 with /dev/nvme2n1... 2024-03-10T03:48:15.082 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph-volume lvm zap /dev/nvme2n1 2024-03-10T03:48:15.089 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[31842]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-10T03:48:15.089 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[31842]: osdmap e37: 6 total, 5 up, 6 in 2024-03-10T03:48:15.090 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T03:48:15.090 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[31842]: from='osd.5 [v2:172.21.15.71:6808/3012063760,v1:172.21.15.71:6809/3012063760]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-10T03:48:15.090 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[31842]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-10T03:48:15.090 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:15.090 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:15.090 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:48:15.090 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:48:15.090 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:48:15.090 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[31842]: Adjusting osd_memory_target on smithi071 to 8032M 2024-03-10T03:48:15.090 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:15.091 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:15.091 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:48:15.091 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:15.091 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[31842]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-10T03:48:15.091 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[25256]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-10T03:48:15.091 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[25256]: osdmap e37: 6 total, 5 up, 6 in 2024-03-10T03:48:15.091 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T03:48:15.091 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[25256]: from='osd.5 [v2:172.21.15.71:6808/3012063760,v1:172.21.15.71:6809/3012063760]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-10T03:48:15.091 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[25256]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-10T03:48:15.091 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:15.092 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:15.092 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:48:15.092 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:48:15.092 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:48:15.092 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[25256]: Adjusting osd_memory_target on smithi071 to 8032M 2024-03-10T03:48:15.092 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:15.092 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:15.092 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:48:15.092 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:15.092 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:15 smithi006 ceph-mon[25256]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-10T03:48:15.368 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T03:48:15.754 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:48:15 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:48:15.485+0000 7ff6faf6d700 -1 osd.5 0 waiting for initial osdmap 2024-03-10T03:48:15.754 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:48:15 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:48:15.489+0000 7ff6f2d58700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-10T03:48:16.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:16 smithi006 ceph-mon[25256]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-03-10T03:48:16.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:16 smithi006 ceph-mon[25256]: osdmap e38: 6 total, 5 up, 6 in 2024-03-10T03:48:16.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:16 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T03:48:16.338 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:16 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T03:48:16.338 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:16 smithi006 ceph-mon[25256]: from='osd.5 ' entity='osd.5' 2024-03-10T03:48:16.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:16 smithi006 ceph-mon[31842]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-03-10T03:48:16.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:16 smithi006 ceph-mon[31842]: osdmap e38: 6 total, 5 up, 6 in 2024-03-10T03:48:16.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:16 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T03:48:16.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:16 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T03:48:16.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:16 smithi006 ceph-mon[31842]: from='osd.5 ' entity='osd.5' 2024-03-10T03:48:16.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:16 smithi071 ceph-mon[27181]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-03-10T03:48:16.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:16 smithi071 ceph-mon[27181]: osdmap e38: 6 total, 5 up, 6 in 2024-03-10T03:48:16.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:16 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T03:48:16.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:16 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T03:48:16.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:16 smithi071 ceph-mon[27181]: from='osd.5 ' entity='osd.5' 2024-03-10T03:48:17.246 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping: /dev/nvme2n1 2024-03-10T03:48:17.247 INFO:teuthology.orchestra.run.smithi071.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-10T03:48:17.247 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme2n1 bs=1M count=10 conv=fsync 2024-03-10T03:48:17.247 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10+0 records in 2024-03-10T03:48:17.247 INFO:teuthology.orchestra.run.smithi071.stderr:10+0 records out 2024-03-10T03:48:17.247 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129492 s, 810 MB/s 2024-03-10T03:48:17.247 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping successful for: 2024-03-10T03:48:17.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:17 smithi006 ceph-mon[25256]: purged_snaps scrub starts 2024-03-10T03:48:17.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:17 smithi006 ceph-mon[25256]: purged_snaps scrub ok 2024-03-10T03:48:17.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:17 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T03:48:17.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:17 smithi006 ceph-mon[25256]: osd.5 [v2:172.21.15.71:6808/3012063760,v1:172.21.15.71:6809/3012063760] boot 2024-03-10T03:48:17.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:17 smithi006 ceph-mon[25256]: osdmap e39: 6 total, 6 up, 6 in 2024-03-10T03:48:17.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:17 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T03:48:17.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:17 smithi006 ceph-mon[25256]: pgmap v141: 1 pgs: 1 unknown; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-10T03:48:17.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:17 smithi006 ceph-mon[31842]: purged_snaps scrub starts 2024-03-10T03:48:17.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:17 smithi006 ceph-mon[31842]: purged_snaps scrub ok 2024-03-10T03:48:17.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:17 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T03:48:17.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:17 smithi006 ceph-mon[31842]: osd.5 [v2:172.21.15.71:6808/3012063760,v1:172.21.15.71:6809/3012063760] boot 2024-03-10T03:48:17.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:17 smithi006 ceph-mon[31842]: osdmap e39: 6 total, 6 up, 6 in 2024-03-10T03:48:17.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:17 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T03:48:17.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:17 smithi006 ceph-mon[31842]: pgmap v141: 1 pgs: 1 unknown; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-10T03:48:17.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:17 smithi071 ceph-mon[27181]: purged_snaps scrub starts 2024-03-10T03:48:17.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:17 smithi071 ceph-mon[27181]: purged_snaps scrub ok 2024-03-10T03:48:17.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:17 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T03:48:17.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:17 smithi071 ceph-mon[27181]: osd.5 [v2:172.21.15.71:6808/3012063760,v1:172.21.15.71:6809/3012063760] boot 2024-03-10T03:48:17.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:17 smithi071 ceph-mon[27181]: osdmap e39: 6 total, 6 up, 6 in 2024-03-10T03:48:17.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:17 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T03:48:17.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:17 smithi071 ceph-mon[27181]: pgmap v141: 1 pgs: 1 unknown; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-10T03:48:17.789 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph orch daemon add osd smithi071:/dev/nvme2n1 2024-03-10T03:48:18.023 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T03:48:18.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:18 smithi071 ceph-mon[27181]: osdmap e40: 6 total, 6 up, 6 in 2024-03-10T03:48:18.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:18 smithi006 ceph-mon[25256]: osdmap e40: 6 total, 6 up, 6 in 2024-03-10T03:48:18.349 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:18 smithi006 ceph-mon[31842]: osdmap e40: 6 total, 6 up, 6 in 2024-03-10T03:48:19.252 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:19 smithi071 ceph-mon[27181]: osdmap e41: 6 total, 6 up, 6 in 2024-03-10T03:48:19.252 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:19 smithi071 ceph-mon[27181]: pgmap v144: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:19.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:19 smithi006 ceph-mon[25256]: osdmap e41: 6 total, 6 up, 6 in 2024-03-10T03:48:19.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:19 smithi006 ceph-mon[25256]: pgmap v144: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:19.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:19 smithi006 ceph-mon[31842]: osdmap e41: 6 total, 6 up, 6 in 2024-03-10T03:48:19.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:19 smithi006 ceph-mon[31842]: pgmap v144: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:20.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:20 smithi006 ceph-mon[25256]: osdmap e42: 6 total, 6 up, 6 in 2024-03-10T03:48:20.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:20 smithi006 ceph-mon[25256]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:48:20.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:20 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T03:48:20.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:20 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T03:48:20.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:20 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:20.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:20 smithi006 ceph-mon[31842]: osdmap e42: 6 total, 6 up, 6 in 2024-03-10T03:48:20.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:20 smithi006 ceph-mon[31842]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:48:20.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:20 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T03:48:20.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:20 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T03:48:20.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:20 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:20.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:20 smithi071 ceph-mon[27181]: osdmap e42: 6 total, 6 up, 6 in 2024-03-10T03:48:20.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:20 smithi071 ceph-mon[27181]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:/dev/nvme2n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:48:20.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:20 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T03:48:20.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:20 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T03:48:20.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:20 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:21.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:21 smithi006 ceph-mon[25256]: osdmap e43: 6 total, 6 up, 6 in 2024-03-10T03:48:21.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:21.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:21.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:21 smithi006 ceph-mon[25256]: Detected new or changed devices on smithi071 2024-03-10T03:48:21.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:21.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:48:21.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:21.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:48:21.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:21 smithi006 ceph-mon[25256]: pgmap v147: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:21.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:21.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:21 smithi006 ceph-mon[31842]: osdmap e43: 6 total, 6 up, 6 in 2024-03-10T03:48:21.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:21.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:21.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:21 smithi006 ceph-mon[31842]: Detected new or changed devices on smithi071 2024-03-10T03:48:21.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:21.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:48:21.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:21.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:48:21.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:21 smithi006 ceph-mon[31842]: pgmap v147: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:21.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:21.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:21 smithi071 ceph-mon[27181]: osdmap e43: 6 total, 6 up, 6 in 2024-03-10T03:48:21.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:21.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:21.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:21 smithi071 ceph-mon[27181]: Detected new or changed devices on smithi071 2024-03-10T03:48:21.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:21.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:48:21.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:21.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:48:21.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:21 smithi071 ceph-mon[27181]: pgmap v147: 1 pgs: 1 unknown; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:21.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:22.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:22 smithi006 ceph-mon[25256]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-10T03:48:22.336 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:22 smithi006 ceph-mon[31842]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-10T03:48:22.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:22 smithi071 ceph-mon[27181]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-10T03:48:24.086 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[31842]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-10T03:48:24.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:24.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:24.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:24.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:48:24.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:24.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:48:24.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[31842]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-10T03:48:24.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:24.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[31842]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-03-10T03:48:24.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[31842]: Cluster is now healthy 2024-03-10T03:48:24.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[31842]: from='client.? 172.21.15.71:0/4143193176' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e07f4f7-b672-4a77-abe5-3a440eff844f"}]: dispatch 2024-03-10T03:48:24.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[31842]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e07f4f7-b672-4a77-abe5-3a440eff844f"}]: dispatch 2024-03-10T03:48:24.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[31842]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e07f4f7-b672-4a77-abe5-3a440eff844f"}]': finished 2024-03-10T03:48:24.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[31842]: osdmap e44: 7 total, 6 up, 7 in 2024-03-10T03:48:24.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T03:48:24.088 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[25256]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-10T03:48:24.088 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:24.088 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:24.088 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:24.088 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:48:24.088 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:24.088 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:48:24.088 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[25256]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-10T03:48:24.088 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:24.089 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[25256]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-03-10T03:48:24.089 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[25256]: Cluster is now healthy 2024-03-10T03:48:24.089 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[25256]: from='client.? 172.21.15.71:0/4143193176' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e07f4f7-b672-4a77-abe5-3a440eff844f"}]: dispatch 2024-03-10T03:48:24.089 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[25256]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e07f4f7-b672-4a77-abe5-3a440eff844f"}]: dispatch 2024-03-10T03:48:24.089 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[25256]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e07f4f7-b672-4a77-abe5-3a440eff844f"}]': finished 2024-03-10T03:48:24.089 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[25256]: osdmap e44: 7 total, 6 up, 7 in 2024-03-10T03:48:24.089 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:23 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T03:48:24.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:23 smithi071 ceph-mon[27181]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-10T03:48:24.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:24.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:24.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:24.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:48:24.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:24.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:48:24.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:23 smithi071 ceph-mon[27181]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-03-10T03:48:24.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:24.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:23 smithi071 ceph-mon[27181]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-03-10T03:48:24.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:23 smithi071 ceph-mon[27181]: Cluster is now healthy 2024-03-10T03:48:24.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:23 smithi071 ceph-mon[27181]: from='client.? 172.21.15.71:0/4143193176' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e07f4f7-b672-4a77-abe5-3a440eff844f"}]: dispatch 2024-03-10T03:48:24.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:23 smithi071 ceph-mon[27181]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e07f4f7-b672-4a77-abe5-3a440eff844f"}]: dispatch 2024-03-10T03:48:24.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:23 smithi071 ceph-mon[27181]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e07f4f7-b672-4a77-abe5-3a440eff844f"}]': finished 2024-03-10T03:48:24.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:23 smithi071 ceph-mon[27181]: osdmap e44: 7 total, 6 up, 7 in 2024-03-10T03:48:24.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:23 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T03:48:25.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:24 smithi006 ceph-mon[25256]: from='client.? 172.21.15.71:0/4247651208' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T03:48:25.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:24 smithi006 ceph-mon[31842]: from='client.? 172.21.15.71:0/4247651208' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T03:48:25.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:24 smithi071 ceph-mon[27181]: from='client.? 172.21.15.71:0/4247651208' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T03:48:26.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:25 smithi006 ceph-mon[25256]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-10T03:48:26.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:25 smithi006 ceph-mon[31842]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-10T03:48:26.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:25 smithi071 ceph-mon[27181]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-10T03:48:28.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:27 smithi006 ceph-mon[25256]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-10T03:48:28.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:27 smithi006 ceph-mon[31842]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-10T03:48:28.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:27 smithi071 ceph-mon[27181]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-10T03:48:30.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:29 smithi006 ceph-mon[25256]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-10T03:48:30.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:29 smithi006 ceph-mon[31842]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-10T03:48:30.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:29 smithi071 ceph-mon[27181]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-10T03:48:32.024 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:31 smithi071 ceph-mon[27181]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:32.025 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:31 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:32.025 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:31 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:32.025 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:31 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:32.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:31 smithi006 ceph-mon[25256]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:32.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:31 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:32.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:31 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:32.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:31 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:32.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:31 smithi006 ceph-mon[31842]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:32.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:31 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:32.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:31 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:32.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:31 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:32.944 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:32 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-10T03:48:32.944 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:32 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:33.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:32 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-10T03:48:33.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:32 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:33.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:32 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-10T03:48:33.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:32 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:33.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:33 smithi071 ceph-mon[27181]: Deploying daemon osd.6 on smithi071 2024-03-10T03:48:33.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:33 smithi071 ceph-mon[27181]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:34.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:33 smithi006 ceph-mon[25256]: Deploying daemon osd.6 on smithi071 2024-03-10T03:48:34.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:33 smithi006 ceph-mon[25256]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:34.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:33 smithi006 ceph-mon[31842]: Deploying daemon osd.6 on smithi071 2024-03-10T03:48:34.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:33 smithi006 ceph-mon[31842]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:36.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:35 smithi006 ceph-mon[25256]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:36.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:35 smithi006 ceph-mon[31842]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:36.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:35 smithi071 ceph-mon[27181]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:38.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:37 smithi006 ceph-mon[25256]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:38.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:37 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:38.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:37 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:38.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:37 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:48:38.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:37 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:38.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:37 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:48:38.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:37 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:38.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:37 smithi006 ceph-mon[31842]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:38.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:37 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:38.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:37 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:38.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:37 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:48:38.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:37 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:38.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:37 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:48:38.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:37 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:38.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:37 smithi071 ceph-mon[27181]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:38.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:37 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:38.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:37 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:38.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:37 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:48:38.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:37 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:38.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:37 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:48:38.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:37 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:40.066 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:39 smithi071 ceph-mon[27181]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:40.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:39 smithi006 ceph-mon[25256]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:40.086 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:39 smithi006 ceph-mon[31842]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:40.776 INFO:teuthology.orchestra.run.smithi071.stdout:Created osd(s) 6 on host 'smithi071' 2024-03-10T03:48:41.072 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:40 smithi071 ceph-mon[27181]: from='osd.6 [v2:172.21.15.71:6816/2652028816,v1:172.21.15.71:6817/2652028816]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-10T03:48:41.072 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:40 smithi071 ceph-mon[27181]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-10T03:48:41.072 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:40 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:41.072 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:40 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:41.072 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:40 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:48:41.072 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:40 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:48:41.072 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:40 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:48:41.072 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:40 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:48:41.072 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:40 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:41.073 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:40 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:41.073 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:40 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:48:41.073 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:40 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:41.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:40 smithi006 ceph-mon[25256]: from='osd.6 [v2:172.21.15.71:6816/2652028816,v1:172.21.15.71:6817/2652028816]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-10T03:48:41.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:40 smithi006 ceph-mon[25256]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-10T03:48:41.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:40 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:41.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:40 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:41.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:40 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:48:41.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:40 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:48:41.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:40 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:48:41.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:40 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:48:41.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:40 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:41.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:40 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:41.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:40 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:48:41.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:40 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:41.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:40 smithi006 ceph-mon[31842]: from='osd.6 [v2:172.21.15.71:6816/2652028816,v1:172.21.15.71:6817/2652028816]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-10T03:48:41.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:40 smithi006 ceph-mon[31842]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-10T03:48:41.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:40 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:41.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:40 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:41.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:40 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:48:41.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:40 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:48:41.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:40 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:48:41.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:40 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:48:41.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:40 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:41.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:40 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:41.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:40 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:48:41.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:40 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:41.372 DEBUG:teuthology.orchestra.run.smithi071:osd.6> sudo journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.6.service 2024-03-10T03:48:41.374 INFO:tasks.cephadm:Deploying osd.7 on smithi071 with /dev/nvme1n1... 2024-03-10T03:48:41.375 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph-volume lvm zap /dev/nvme1n1 2024-03-10T03:48:41.659 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T03:48:41.992 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:41 smithi071 ceph-mon[27181]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:41.992 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:41 smithi071 ceph-mon[27181]: Adjusting osd_memory_target on smithi071 to 5354M 2024-03-10T03:48:41.992 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:41 smithi071 ceph-mon[27181]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-10T03:48:41.993 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:41 smithi071 ceph-mon[27181]: osdmap e45: 7 total, 6 up, 7 in 2024-03-10T03:48:41.993 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:41 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T03:48:41.993 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:41 smithi071 ceph-mon[27181]: from='osd.6 [v2:172.21.15.71:6816/2652028816,v1:172.21.15.71:6817/2652028816]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-10T03:48:41.993 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:41 smithi071 ceph-mon[27181]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-10T03:48:42.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:41 smithi006 ceph-mon[25256]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:42.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:41 smithi006 ceph-mon[25256]: Adjusting osd_memory_target on smithi071 to 5354M 2024-03-10T03:48:42.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:41 smithi006 ceph-mon[25256]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-10T03:48:42.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:41 smithi006 ceph-mon[25256]: osdmap e45: 7 total, 6 up, 7 in 2024-03-10T03:48:42.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:41 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T03:48:42.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:41 smithi006 ceph-mon[25256]: from='osd.6 [v2:172.21.15.71:6816/2652028816,v1:172.21.15.71:6817/2652028816]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-10T03:48:42.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:41 smithi006 ceph-mon[25256]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-10T03:48:42.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:41 smithi006 ceph-mon[31842]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:42.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:41 smithi006 ceph-mon[31842]: Adjusting osd_memory_target on smithi071 to 5354M 2024-03-10T03:48:42.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:41 smithi006 ceph-mon[31842]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-10T03:48:42.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:41 smithi006 ceph-mon[31842]: osdmap e45: 7 total, 6 up, 7 in 2024-03-10T03:48:42.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:41 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T03:48:42.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:41 smithi006 ceph-mon[31842]: from='osd.6 [v2:172.21.15.71:6816/2652028816,v1:172.21.15.71:6817/2652028816]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-10T03:48:42.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:41 smithi006 ceph-mon[31842]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-10T03:48:42.591 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:48:42 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:48:42.311+0000 7fdc00392700 -1 osd.6 0 waiting for initial osdmap 2024-03-10T03:48:42.591 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:48:42 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:48:42.318+0000 7fdbf817d700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-10T03:48:42.978 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:42 smithi071 ceph-mon[27181]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-03-10T03:48:42.979 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:42 smithi071 ceph-mon[27181]: osdmap e46: 7 total, 6 up, 7 in 2024-03-10T03:48:42.979 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:42 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T03:48:42.979 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:42 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T03:48:42.979 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:42 smithi071 ceph-mon[27181]: from='osd.6 ' entity='osd.6' 2024-03-10T03:48:42.979 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:42 smithi071 ceph-mon[27181]: osdmap e47: 7 total, 6 up, 7 in 2024-03-10T03:48:42.979 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:42 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T03:48:42.979 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:42 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T03:48:43.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:42 smithi006 ceph-mon[25256]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-03-10T03:48:43.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:42 smithi006 ceph-mon[25256]: osdmap e46: 7 total, 6 up, 7 in 2024-03-10T03:48:43.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:42 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T03:48:43.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:42 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T03:48:43.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:42 smithi006 ceph-mon[25256]: from='osd.6 ' entity='osd.6' 2024-03-10T03:48:43.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:42 smithi006 ceph-mon[25256]: osdmap e47: 7 total, 6 up, 7 in 2024-03-10T03:48:43.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:42 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T03:48:43.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:42 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T03:48:43.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:42 smithi006 ceph-mon[31842]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-03-10T03:48:43.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:42 smithi006 ceph-mon[31842]: osdmap e46: 7 total, 6 up, 7 in 2024-03-10T03:48:43.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:42 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T03:48:43.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:42 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T03:48:43.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:42 smithi006 ceph-mon[31842]: from='osd.6 ' entity='osd.6' 2024-03-10T03:48:43.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:42 smithi006 ceph-mon[31842]: osdmap e47: 7 total, 6 up, 7 in 2024-03-10T03:48:43.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:42 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T03:48:43.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:42 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T03:48:43.574 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping: /dev/nvme1n1 2024-03-10T03:48:43.574 INFO:teuthology.orchestra.run.smithi071.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-10T03:48:43.574 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme1n1 bs=1M count=10 conv=fsync 2024-03-10T03:48:43.574 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10+0 records in 2024-03-10T03:48:43.574 INFO:teuthology.orchestra.run.smithi071.stderr:10+0 records out 2024-03-10T03:48:43.574 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0144995 s, 723 MB/s 2024-03-10T03:48:43.574 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping successful for: 2024-03-10T03:48:44.179 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:43 smithi071 ceph-mon[27181]: purged_snaps scrub starts 2024-03-10T03:48:44.180 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:43 smithi071 ceph-mon[27181]: purged_snaps scrub ok 2024-03-10T03:48:44.180 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:43 smithi071 ceph-mon[27181]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:44.180 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:43 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:44.180 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:43 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:44.180 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:43 smithi071 ceph-mon[27181]: Detected new or changed devices on smithi071 2024-03-10T03:48:44.180 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:43 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:44.180 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:43 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:48:44.180 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:43 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:44.180 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:43 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:48:44.180 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:43 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:44.180 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:43 smithi071 ceph-mon[27181]: osd.6 [v2:172.21.15.71:6816/2652028816,v1:172.21.15.71:6817/2652028816] boot 2024-03-10T03:48:44.180 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:43 smithi071 ceph-mon[27181]: osdmap e48: 7 total, 7 up, 7 in 2024-03-10T03:48:44.180 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:43 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T03:48:44.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[25256]: purged_snaps scrub starts 2024-03-10T03:48:44.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[25256]: purged_snaps scrub ok 2024-03-10T03:48:44.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[25256]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:44.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:44.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:44.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[25256]: Detected new or changed devices on smithi071 2024-03-10T03:48:44.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:44.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:48:44.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:44.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:48:44.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:44.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[25256]: osd.6 [v2:172.21.15.71:6816/2652028816,v1:172.21.15.71:6817/2652028816] boot 2024-03-10T03:48:44.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[25256]: osdmap e48: 7 total, 7 up, 7 in 2024-03-10T03:48:44.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T03:48:44.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[31842]: purged_snaps scrub starts 2024-03-10T03:48:44.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[31842]: purged_snaps scrub ok 2024-03-10T03:48:44.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[31842]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-10T03:48:44.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:44.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:44.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[31842]: Detected new or changed devices on smithi071 2024-03-10T03:48:44.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:44.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:48:44.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:44.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:48:44.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:44.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[31842]: osd.6 [v2:172.21.15.71:6816/2652028816,v1:172.21.15.71:6817/2652028816] boot 2024-03-10T03:48:44.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[31842]: osdmap e48: 7 total, 7 up, 7 in 2024-03-10T03:48:44.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:43 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T03:48:44.541 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph orch daemon add osd smithi071:/dev/nvme1n1 2024-03-10T03:48:44.794 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T03:48:45.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:45 smithi071 ceph-mon[27181]: osdmap e49: 7 total, 7 up, 7 in 2024-03-10T03:48:45.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:45 smithi071 ceph-mon[27181]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:48:45.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:45 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:45.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:45 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:45.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:45 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:45.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:45 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:48:45.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:45 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:45.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:45 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:48:45.592 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:45 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:45.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:45 smithi006 ceph-mon[25256]: osdmap e49: 7 total, 7 up, 7 in 2024-03-10T03:48:45.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:45 smithi006 ceph-mon[25256]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:48:45.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:45 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:45.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:45 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:45.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:45 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:45.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:45 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:48:45.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:45 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:45.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:45 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:48:45.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:45 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:45.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:45 smithi006 ceph-mon[31842]: osdmap e49: 7 total, 7 up, 7 in 2024-03-10T03:48:45.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:45 smithi006 ceph-mon[31842]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:48:45.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:45 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:45.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:45 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:45.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:45 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:45.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:45 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:48:45.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:45 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:45.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:45 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:48:45.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:45 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:46.565 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:46 smithi071 ceph-mon[27181]: osdmap e50: 7 total, 7 up, 7 in 2024-03-10T03:48:46.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:46 smithi006 ceph-mon[25256]: osdmap e50: 7 total, 7 up, 7 in 2024-03-10T03:48:46.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:46 smithi006 ceph-mon[31842]: osdmap e50: 7 total, 7 up, 7 in 2024-03-10T03:48:47.414 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:47 smithi071 ceph-mon[27181]: osdmap e51: 7 total, 7 up, 7 in 2024-03-10T03:48:47.415 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:47 smithi071 ceph-mon[27181]: pgmap v169: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-10T03:48:47.415 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:47 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T03:48:47.415 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:47 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T03:48:47.415 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:47 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:47.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:47 smithi006 ceph-mon[25256]: osdmap e51: 7 total, 7 up, 7 in 2024-03-10T03:48:47.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:47 smithi006 ceph-mon[25256]: pgmap v169: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-10T03:48:47.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:47 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T03:48:47.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:47 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T03:48:47.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:47 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:47.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:47 smithi006 ceph-mon[31842]: osdmap e51: 7 total, 7 up, 7 in 2024-03-10T03:48:47.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:47 smithi006 ceph-mon[31842]: pgmap v169: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-10T03:48:47.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:47 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-10T03:48:47.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:47 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-10T03:48:47.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:47 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:48.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:48 smithi071 ceph-mon[27181]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:48:48.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:48 smithi071 ceph-mon[27181]: osdmap e52: 7 total, 7 up, 7 in 2024-03-10T03:48:48.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:48 smithi006 ceph-mon[25256]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:48:48.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:48 smithi006 ceph-mon[25256]: osdmap e52: 7 total, 7 up, 7 in 2024-03-10T03:48:48.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:48 smithi006 ceph-mon[31842]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:/dev/nvme1n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:48:48.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:48 smithi006 ceph-mon[31842]: osdmap e52: 7 total, 7 up, 7 in 2024-03-10T03:48:49.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:49 smithi006 ceph-mon[25256]: pgmap v171: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-03-10T03:48:49.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:49 smithi006 ceph-mon[31842]: pgmap v171: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-03-10T03:48:49.651 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:49 smithi071 ceph-mon[27181]: pgmap v171: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-03-10T03:48:50.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:50 smithi006 ceph-mon[25256]: from='client.? 172.21.15.71:0/430465842' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaeda1f0-6c59-44a2-b394-ebacf817248c"}]: dispatch 2024-03-10T03:48:50.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:50 smithi006 ceph-mon[25256]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaeda1f0-6c59-44a2-b394-ebacf817248c"}]: dispatch 2024-03-10T03:48:50.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:50 smithi006 ceph-mon[25256]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aaeda1f0-6c59-44a2-b394-ebacf817248c"}]': finished 2024-03-10T03:48:50.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:50 smithi006 ceph-mon[25256]: osdmap e53: 8 total, 7 up, 8 in 2024-03-10T03:48:50.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:50 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T03:48:50.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:50 smithi006 ceph-mon[31842]: from='client.? 172.21.15.71:0/430465842' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaeda1f0-6c59-44a2-b394-ebacf817248c"}]: dispatch 2024-03-10T03:48:50.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:50 smithi006 ceph-mon[31842]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaeda1f0-6c59-44a2-b394-ebacf817248c"}]: dispatch 2024-03-10T03:48:50.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:50 smithi006 ceph-mon[31842]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aaeda1f0-6c59-44a2-b394-ebacf817248c"}]': finished 2024-03-10T03:48:50.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:50 smithi006 ceph-mon[31842]: osdmap e53: 8 total, 7 up, 8 in 2024-03-10T03:48:50.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:50 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T03:48:50.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:50 smithi071 ceph-mon[27181]: from='client.? 172.21.15.71:0/430465842' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaeda1f0-6c59-44a2-b394-ebacf817248c"}]: dispatch 2024-03-10T03:48:50.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:50 smithi071 ceph-mon[27181]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aaeda1f0-6c59-44a2-b394-ebacf817248c"}]: dispatch 2024-03-10T03:48:50.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:50 smithi071 ceph-mon[27181]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aaeda1f0-6c59-44a2-b394-ebacf817248c"}]': finished 2024-03-10T03:48:50.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:50 smithi071 ceph-mon[27181]: osdmap e53: 8 total, 7 up, 8 in 2024-03-10T03:48:50.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:50 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T03:48:52.272 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:51 smithi006 ceph-mon[25256]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-03-10T03:48:52.273 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:51 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:52.273 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:51 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:52.273 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:51 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:52.273 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:51 smithi006 ceph-mon[25256]: from='client.? 172.21.15.71:0/3744322221' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T03:48:52.274 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:51 smithi006 ceph-mon[31842]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-03-10T03:48:52.274 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:51 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:52.274 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:51 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:52.274 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:51 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:52.274 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:51 smithi006 ceph-mon[31842]: from='client.? 172.21.15.71:0/3744322221' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T03:48:52.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:51 smithi071 ceph-mon[27181]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-03-10T03:48:52.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:51 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:52.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:51 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:52.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:51 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:48:52.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:51 smithi071 ceph-mon[27181]: from='client.? 172.21.15.71:0/3744322221' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-10T03:48:54.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:53 smithi006 ceph-mon[25256]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:48:54.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:53 smithi006 ceph-mon[31842]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:48:54.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:53 smithi071 ceph-mon[27181]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:48:56.225 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:55 smithi071 ceph-mon[27181]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:48:56.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:55 smithi006 ceph-mon[25256]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:48:56.336 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:55 smithi006 ceph-mon[31842]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:48:58.232 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:57 smithi071 ceph-mon[27181]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:48:58.232 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:57 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-10T03:48:58.232 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:57 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:58.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:57 smithi006 ceph-mon[25256]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:48:58.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:57 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-10T03:48:58.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:57 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:58.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:57 smithi006 ceph-mon[31842]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:48:58.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:57 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-10T03:48:58.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:57 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:48:59.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:58 smithi071 ceph-mon[27181]: Deploying daemon osd.7 on smithi071 2024-03-10T03:48:59.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:58 smithi006 ceph-mon[25256]: Deploying daemon osd.7 on smithi071 2024-03-10T03:48:59.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:58 smithi006 ceph-mon[31842]: Deploying daemon osd.7 on smithi071 2024-03-10T03:49:00.302 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:48:59 smithi071 ceph-mon[27181]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:49:00.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:48:59 smithi006 ceph-mon[25256]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:49:00.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:48:59 smithi006 ceph-mon[31842]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:49:02.284 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:01 smithi071 ceph-mon[27181]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:49:02.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:01 smithi006 ceph-mon[25256]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:49:02.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:01 smithi006 ceph-mon[31842]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:49:04.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:03 smithi006 ceph-mon[25256]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:49:04.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:03 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:04.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:03 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:04.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:03 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:04.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:03 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:04.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:03 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:04.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:03 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:04.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:03 smithi006 ceph-mon[31842]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:49:04.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:03 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:04.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:03 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:04.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:03 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:04.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:03 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:04.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:03 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:04.088 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:03 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:04.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:03 smithi071 ceph-mon[27181]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:49:04.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:03 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:04.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:03 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:04.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:03 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:04.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:03 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:04.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:03 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:04.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:03 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:06.220 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:05 smithi071 ceph-mon[27181]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:49:06.220 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:05 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:06.221 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:05 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:06.221 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:05 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:06.221 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:05 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:06.221 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:05 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:06.221 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:05 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:06.221 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:05 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:06.221 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:05 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:06.221 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:05 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:06.221 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:05 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:06.221 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:05 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:06.221 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:05 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:06.221 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:05 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:06.221 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:05 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:06.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[25256]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:49:06.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:06.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:06.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:06.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:06.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:06.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:06.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:06.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:06.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:06.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:06.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:06.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:06.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:06.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:06.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[31842]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:49:06.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:06.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:06.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:06.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:06.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:06.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:06.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:06.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:06.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:06.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:06.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:06.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:06.339 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:06.339 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:05 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:07.178 INFO:teuthology.orchestra.run.smithi071.stdout:Created osd(s) 7 on host 'smithi071' 2024-03-10T03:49:07.178 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:06 smithi071 ceph-mon[27181]: from='osd.7 [v2:172.21.15.71:6824/3011457307,v1:172.21.15.71:6825/3011457307]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-10T03:49:07.179 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:06 smithi071 ceph-mon[27181]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-10T03:49:07.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:06 smithi006 ceph-mon[25256]: from='osd.7 [v2:172.21.15.71:6824/3011457307,v1:172.21.15.71:6825/3011457307]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-10T03:49:07.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:06 smithi006 ceph-mon[25256]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-10T03:49:07.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:06 smithi006 ceph-mon[31842]: from='osd.7 [v2:172.21.15.71:6824/3011457307,v1:172.21.15.71:6825/3011457307]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-10T03:49:07.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:06 smithi006 ceph-mon[31842]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-10T03:49:08.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:07 smithi071 ceph-mon[27181]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:49:08.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:07 smithi071 ceph-mon[27181]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-10T03:49:08.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:07 smithi071 ceph-mon[27181]: from='osd.7 [v2:172.21.15.71:6824/3011457307,v1:172.21.15.71:6825/3011457307]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-10T03:49:08.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:07 smithi071 ceph-mon[27181]: osdmap e54: 8 total, 7 up, 8 in 2024-03-10T03:49:08.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:07 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T03:49:08.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:07 smithi071 ceph-mon[27181]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-10T03:49:08.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:07 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:08.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:07 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:08.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:07 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:49:08.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:07 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:08.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:07 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:49:08.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:07 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:49:08.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:07 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:49:08.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:07 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:08.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:07 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:08.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:07 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:08.092 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:07 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:08.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[25256]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:49:08.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[25256]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-10T03:49:08.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[25256]: from='osd.7 [v2:172.21.15.71:6824/3011457307,v1:172.21.15.71:6825/3011457307]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-10T03:49:08.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[25256]: osdmap e54: 8 total, 7 up, 8 in 2024-03-10T03:49:08.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T03:49:08.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[25256]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-10T03:49:08.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:08.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:08.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:49:08.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:08.338 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:49:08.338 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:49:08.338 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:49:08.338 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:08.338 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:08.338 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:08.338 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:08.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[31842]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:49:08.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[31842]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-10T03:49:08.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[31842]: from='osd.7 [v2:172.21.15.71:6824/3011457307,v1:172.21.15.71:6825/3011457307]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-10T03:49:08.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[31842]: osdmap e54: 8 total, 7 up, 8 in 2024-03-10T03:49:08.339 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T03:49:08.339 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[31842]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-03-10T03:49:08.339 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:08.339 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:08.339 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:49:08.339 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:08.339 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:49:08.339 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:49:08.339 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:49:08.339 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:08.339 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:08.339 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:08.339 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:07 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:08.538 DEBUG:teuthology.orchestra.run.smithi071:osd.7> sudo journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.7.service 2024-03-10T03:49:08.541 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-10T03:49:08.541 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph osd stat -f json 2024-03-10T03:49:08.799 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:49:09.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:08 smithi071 ceph-mon[27181]: Adjusting osd_memory_target on smithi071 to 4016M 2024-03-10T03:49:09.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:08 smithi071 ceph-mon[27181]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-03-10T03:49:09.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:08 smithi071 ceph-mon[27181]: osdmap e55: 8 total, 7 up, 8 in 2024-03-10T03:49:09.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:08 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T03:49:09.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:08 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T03:49:09.183 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:08 smithi006 ceph-mon[25256]: Adjusting osd_memory_target on smithi071 to 4016M 2024-03-10T03:49:09.183 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:08 smithi006 ceph-mon[25256]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-03-10T03:49:09.183 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:08 smithi006 ceph-mon[25256]: osdmap e55: 8 total, 7 up, 8 in 2024-03-10T03:49:09.183 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:08 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T03:49:09.183 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:08 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T03:49:09.183 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:08 smithi006 ceph-mon[31842]: Adjusting osd_memory_target on smithi071 to 4016M 2024-03-10T03:49:09.183 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:08 smithi006 ceph-mon[31842]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-03-10T03:49:09.184 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:08 smithi006 ceph-mon[31842]: osdmap e55: 8 total, 7 up, 8 in 2024-03-10T03:49:09.184 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:08 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T03:49:09.184 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:08 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T03:49:10.195 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:09 smithi006 ceph-mon[31842]: purged_snaps scrub starts 2024-03-10T03:49:10.196 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:09 smithi006 ceph-mon[31842]: purged_snaps scrub ok 2024-03-10T03:49:10.196 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:09 smithi006 ceph-mon[31842]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:49:10.196 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:09 smithi006 ceph-mon[31842]: osd.7 [v2:172.21.15.71:6824/3011457307,v1:172.21.15.71:6825/3011457307] boot 2024-03-10T03:49:10.196 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:09 smithi006 ceph-mon[31842]: osdmap e56: 8 total, 8 up, 8 in 2024-03-10T03:49:10.196 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:09 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T03:49:10.196 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:09 smithi006 ceph-mon[31842]: osdmap e57: 8 total, 8 up, 8 in 2024-03-10T03:49:10.196 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:09 smithi006 ceph-mon[25256]: purged_snaps scrub starts 2024-03-10T03:49:10.196 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:09 smithi006 ceph-mon[25256]: purged_snaps scrub ok 2024-03-10T03:49:10.196 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:09 smithi006 ceph-mon[25256]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:49:10.197 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:09 smithi006 ceph-mon[25256]: osd.7 [v2:172.21.15.71:6824/3011457307,v1:172.21.15.71:6825/3011457307] boot 2024-03-10T03:49:10.197 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:09 smithi006 ceph-mon[25256]: osdmap e56: 8 total, 8 up, 8 in 2024-03-10T03:49:10.197 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:09 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T03:49:10.197 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:09 smithi006 ceph-mon[25256]: osdmap e57: 8 total, 8 up, 8 in 2024-03-10T03:49:10.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:09 smithi071 ceph-mon[27181]: purged_snaps scrub starts 2024-03-10T03:49:10.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:09 smithi071 ceph-mon[27181]: purged_snaps scrub ok 2024-03-10T03:49:10.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:09 smithi071 ceph-mon[27181]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-10T03:49:10.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:09 smithi071 ceph-mon[27181]: osd.7 [v2:172.21.15.71:6824/3011457307,v1:172.21.15.71:6825/3011457307] boot 2024-03-10T03:49:10.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:09 smithi071 ceph-mon[27181]: osdmap e56: 8 total, 8 up, 8 in 2024-03-10T03:49:10.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:09 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T03:49:10.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:09 smithi071 ceph-mon[27181]: osdmap e57: 8 total, 8 up, 8 in 2024-03-10T03:49:10.857 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:49:11.169 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:10 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/139687421' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-10T03:49:11.169 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:10 smithi006 ceph-mon[25256]: osdmap e58: 8 total, 8 up, 8 in 2024-03-10T03:49:11.169 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:10 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/139687421' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-10T03:49:11.169 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:10 smithi006 ceph-mon[31842]: osdmap e58: 8 total, 8 up, 8 in 2024-03-10T03:49:11.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:10 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/139687421' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-10T03:49:11.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:10 smithi071 ceph-mon[27181]: osdmap e58: 8 total, 8 up, 8 in 2024-03-10T03:49:11.573 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":57,"num_osds":8,"num_up_osds":8,"osd_up_since":1710042548,"num_in_osds":8,"osd_in_since":1710042530,"num_remapped_pgs":1} 2024-03-10T03:49:11.573 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph osd dump --format=json 2024-03-10T03:49:12.086 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:11 smithi006 ceph-mon[31842]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-10T03:49:12.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:11 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:12.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:11 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:12.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:11 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:12.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:11 smithi006 ceph-mon[25256]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-10T03:49:12.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:11 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:12.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:11 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:12.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:11 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:12.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:11 smithi071 ceph-mon[27181]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-10T03:49:12.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:11 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:12.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:11 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:12.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:11 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:12.741 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:49:13.010 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:12 smithi006 ceph-mon[25256]: osdmap e59: 8 total, 8 up, 8 in 2024-03-10T03:49:13.010 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:12 smithi006 ceph-mon[31842]: osdmap e59: 8 total, 8 up, 8 in 2024-03-10T03:49:13.242 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:12 smithi071 ceph-mon[27181]: osdmap e59: 8 total, 8 up, 8 in 2024-03-10T03:49:14.086 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:13 smithi006 ceph-mon[31842]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-10T03:49:14.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:13 smithi006 ceph-mon[31842]: osdmap e60: 8 total, 8 up, 8 in 2024-03-10T03:49:14.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:13 smithi006 ceph-mon[25256]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-10T03:49:14.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:13 smithi006 ceph-mon[25256]: osdmap e60: 8 total, 8 up, 8 in 2024-03-10T03:49:14.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:13 smithi071 ceph-mon[27181]: pgmap v190: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-10T03:49:14.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:13 smithi071 ceph-mon[27181]: osdmap e60: 8 total, 8 up, 8 in 2024-03-10T03:49:15.702 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:49:15.703 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":60,"fsid":"414ba1e0-de90-11ee-95c7-87774f69a715","created":"2024-03-10T03:43:29.585078+0000","modified":"2024-03-10T03:49:12.953107+0000","last_up_change":"2024-03-10T03:49:08.927831+0000","last_in_change":"2024-03-10T03:48:50.519693+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-10T03:47:01.225820+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"58176ab1-73a2-4d74-8202-a581cd7e934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":726018156},{"type":"v1","addr":"172.21.15.6:6803","nonce":726018156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":726018156},{"type":"v1","addr":"172.21.15.6:6805","nonce":726018156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":726018156},{"type":"v1","addr":"172.21.15.6:6809","nonce":726018156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":726018156},{"type":"v1","addr":"172.21.15.6:6807","nonce":726018156}]},"public_addr":"172.21.15.6:6803/726018156","cluster_addr":"172.21.15.6:6805/726018156","heartbeat_back_addr":"172.21.15.6:6809/726018156","heartbeat_front_addr":"172.21.15.6:6807/726018156","state":["exists","up"]},{"osd":1,"uuid":"b4eee59a-8e2b-4d51-8f0e-feb6600222b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":399164807},{"type":"v1","addr":"172.21.15.6:6811","nonce":399164807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":399164807},{"type":"v1","addr":"172.21.15.6:6813","nonce":399164807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":399164807},{"type":"v1","addr":"172.21.15.6:6817","nonce":399164807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":399164807},{"type":"v1","addr":"172.21.15.6:6815","nonce":399164807}]},"public_addr":"172.21.15.6:6811/399164807","cluster_addr":"172.21.15.6:6813/399164807","heartbeat_back_addr":"172.21.15.6:6817/399164807","heartbeat_front_addr":"172.21.15.6:6815/399164807","state":["exists","up"]},{"osd":2,"uuid":"eadadcfe-1185-4ea7-9eee-cb2c346b70b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3078148920},{"type":"v1","addr":"172.21.15.6:6819","nonce":3078148920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3078148920},{"type":"v1","addr":"172.21.15.6:6821","nonce":3078148920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3078148920},{"type":"v1","addr":"172.21.15.6:6825","nonce":3078148920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3078148920},{"type":"v1","addr":"172.21.15.6:6823","nonce":3078148920}]},"public_addr":"172.21.15.6:6819/3078148920","cluster_addr":"172.21.15.6:6821/3078148920","heartbeat_back_addr":"172.21.15.6:6825/3078148920","heartbeat_front_addr":"172.21.15.6:6823/3078148920","state":["exists","up"]},{"osd":3,"uuid":"ecb7b3ca-493f-4837-9772-146a4f323ba4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":388589465},{"type":"v1","addr":"172.21.15.6:6827","nonce":388589465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":388589465},{"type":"v1","addr":"172.21.15.6:6829","nonce":388589465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":388589465},{"type":"v1","addr":"172.21.15.6:6833","nonce":388589465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":388589465},{"type":"v1","addr":"172.21.15.6:6831","nonce":388589465}]},"public_addr":"172.21.15.6:6827/388589465","cluster_addr":"172.21.15.6:6829/388589465","heartbeat_back_addr":"172.21.15.6:6833/388589465","heartbeat_front_addr":"172.21.15.6:6831/388589465","state":["exists","up"]},{"osd":4,"uuid":"5cdd2e4a-82c4-422a-a716-1ac8527991a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":150275056},{"type":"v1","addr":"172.21.15.71:6801","nonce":150275056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":150275056},{"type":"v1","addr":"172.21.15.71:6803","nonce":150275056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":150275056},{"type":"v1","addr":"172.21.15.71:6807","nonce":150275056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":150275056},{"type":"v1","addr":"172.21.15.71:6805","nonce":150275056}]},"public_addr":"172.21.15.71:6801/150275056","cluster_addr":"172.21.15.71:6803/150275056","heartbeat_back_addr":"172.21.15.71:6807/150275056","heartbeat_front_addr":"172.21.15.71:6805/150275056","state":["exists","up"]},{"osd":5,"uuid":"c1f68af1-e235-4363-8e31-19ea142661fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":3012063760},{"type":"v1","addr":"172.21.15.71:6809","nonce":3012063760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3012063760},{"type":"v1","addr":"172.21.15.71:6811","nonce":3012063760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":3012063760},{"type":"v1","addr":"172.21.15.71:6815","nonce":3012063760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":3012063760},{"type":"v1","addr":"172.21.15.71:6813","nonce":3012063760}]},"public_addr":"172.21.15.71:6809/3012063760","cluster_addr":"172.21.15.71:6811/3012063760","heartbeat_back_addr":"172.21.15.71:6815/3012063760","heartbeat_front_addr":"172.21.15.71:6813/3012063760","state":["exists","up"]},{"osd":6,"uuid":"8e07f4f7-b672-4a77-abe5-3a440eff844f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2652028816},{"type":"v1","addr":"172.21.15.71:6817","nonce":2652028816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2652028816},{"type":"v1","addr":"172.21.15.71:6819","nonce":2652028816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2652028816},{"type":"v1","addr":"172.21.15.71:6823","nonce":2652028816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2652028816},{"type":"v1","addr":"172.21.15.71:6821","nonce":2652028816}]},"public_addr":"172.21.15.71:6817/2652028816","cluster_addr":"172.21.15.71:6819/2652028816","heartbeat_back_addr":"172.21.15.71:6823/2652028816","heartbeat_front_addr":"172.21.15.71:6821/2652028816","state":["exists","up"]},{"osd":7,"uuid":"aaeda1f0-6c59-44a2-b394-ebacf817248c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3011457307},{"type":"v1","addr":"172.21.15.71:6825","nonce":3011457307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":3011457307},{"type":"v1","addr":"172.21.15.71:6827","nonce":3011457307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":3011457307},{"type":"v1","addr":"172.21.15.71:6831","nonce":3011457307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":3011457307},{"type":"v1","addr":"172.21.15.71:6829","nonce":3011457307}]},"public_addr":"172.21.15.71:6825/3011457307","cluster_addr":"172.21.15.71:6827/3011457307","heartbeat_back_addr":"172.21.15.71:6831/3011457307","heartbeat_front_addr":"172.21.15.71:6829/3011457307","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T03:46:03.779380+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T03:46:30.107772+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T03:46:57.905469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T03:47:25.581989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T03:47:49.843184+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T03:48:14.623236+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T03:48:41.677581+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T03:49:07.269732+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:0/1227130243":"2024-03-11T03:44:32.888833+0000","172.21.15.6:6801/194403410":"2024-03-11T03:44:32.888833+0000","172.21.15.6:6800/194403410":"2024-03-11T03:44:32.888833+0000","172.21.15.6:0/3339850563":"2024-03-11T03:44:32.888833+0000","172.21.15.6:0/446417578":"2024-03-11T03:44:05.341856+0000","172.21.15.6:0/437052514":"2024-03-11T03:44:05.341856+0000","172.21.15.6:6800/3433712073":"2024-03-11T03:44:05.341856+0000","172.21.15.6:6801/3433712073":"2024-03-11T03:44:05.341856+0000","172.21.15.6:0/1672617939":"2024-03-11T03:44:05.341856+0000","172.21.15.6:0/2959683900":"2024-03-11T03:44:32.888833+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-10T03:49:16.306 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:15 smithi006 ceph-mon[25256]: pgmap v192: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-10T03:49:16.307 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:15 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/2335912337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-10T03:49:16.307 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:15 smithi006 ceph-mon[31842]: pgmap v192: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-10T03:49:16.307 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:15 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/2335912337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-10T03:49:16.334 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-10T03:47:01.225820+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-10T03:49:16.334 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-10T03:49:16.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:15 smithi071 ceph-mon[27181]: pgmap v192: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-10T03:49:16.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:15 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/2335912337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-10T03:49:16.576 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:49:18.204 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:17 smithi006 ceph-mon[25256]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-10T03:49:18.204 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:17 smithi006 ceph-mon[31842]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-10T03:49:18.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:17 smithi071 ceph-mon[27181]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-10T03:49:18.519 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-10T03:49:19.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:18 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/3849005965' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-10T03:49:19.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:18 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/3849005965' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-10T03:49:19.152 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi006 2024-03-10T03:49:19.153 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi006=foo.a' 2024-03-10T03:49:19.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:18 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/3849005965' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-10T03:49:19.403 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T03:49:20.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:19 smithi071 ceph-mon[27181]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-10T03:49:20.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:19 smithi006 ceph-mon[25256]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-10T03:49:20.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:19 smithi006 ceph-mon[31842]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-10T03:49:21.157 INFO:teuthology.orchestra.run.smithi071.stdout:Scheduled rgw.foo.a update... 2024-03-10T03:49:21.722 DEBUG:teuthology.orchestra.run.smithi006:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@rgw.foo.a.service 2024-03-10T03:49:21.724 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi071 2024-03-10T03:49:21.724 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-10T03:49:21.959 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T03:49:21.972 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:21 smithi071 ceph-mon[27181]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-10T03:49:21.972 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:21.972 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:21.973 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:21.973 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:21.973 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:21.973 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-10T03:49:21.973 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-10T03:49:21.973 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:21.973 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:21 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:22.082 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:21 smithi006 ceph-mon[25256]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-10T03:49:22.083 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:22.083 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:22.083 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:22.083 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:22.083 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:22.083 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-10T03:49:22.083 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-10T03:49:22.083 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:22.083 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:21 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:22.084 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:21 smithi006 ceph-mon[31842]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-10T03:49:22.084 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:22.084 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:22.084 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:22.084 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:22.084 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:22.084 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-10T03:49:22.084 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-10T03:49:22.084 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:22.084 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:21 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:23.126 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:22 smithi006 ceph-mon[25256]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi006=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:49:23.126 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:22 smithi006 ceph-mon[25256]: Saving service rgw.foo.a spec with placement smithi006=foo.a;count:1 2024-03-10T03:49:23.126 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:22 smithi006 ceph-mon[25256]: Deploying daemon rgw.foo.a on smithi006 2024-03-10T03:49:23.127 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:22 smithi006 ceph-mon[31842]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi006=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:49:23.127 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:22 smithi006 ceph-mon[31842]: Saving service rgw.foo.a spec with placement smithi006=foo.a;count:1 2024-03-10T03:49:23.127 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:22 smithi006 ceph-mon[31842]: Deploying daemon rgw.foo.a on smithi006 2024-03-10T03:49:23.230 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:22 smithi071 ceph-mon[27181]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi006=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:49:23.231 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:22 smithi071 ceph-mon[27181]: Saving service rgw.foo.a spec with placement smithi006=foo.a;count:1 2024-03-10T03:49:23.231 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:22 smithi071 ceph-mon[27181]: Deploying daemon rgw.foo.a on smithi006 2024-03-10T03:49:24.086 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 10 03:49:24 smithi006 systemd[1]: Started Ceph rgw.foo.a for 414ba1e0-de90-11ee-95c7-87774f69a715. 2024-03-10T03:49:24.205 INFO:teuthology.orchestra.run.smithi071.stderr:pool 'datapool' created 2024-03-10T03:49:24.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:24 smithi071 ceph-mon[27181]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-10T03:49:24.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:24 smithi071 ceph-mon[27181]: from='client.? 172.21.15.71:0/1960093832' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-10T03:49:24.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:24 smithi071 ceph-mon[27181]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-10T03:49:24.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:24 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:24.559 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:24 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:24.560 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:24 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:24.586 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:24 smithi006 ceph-mon[31842]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-10T03:49:24.586 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:24 smithi006 ceph-mon[31842]: from='client.? 172.21.15.71:0/1960093832' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-10T03:49:24.586 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:24 smithi006 ceph-mon[31842]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-10T03:49:24.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:24 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:24.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:24 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:24.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:24 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:24.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:24 smithi006 ceph-mon[25256]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-10T03:49:24.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:24 smithi006 ceph-mon[25256]: from='client.? 172.21.15.71:0/1960093832' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-10T03:49:24.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:24 smithi006 ceph-mon[25256]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-10T03:49:24.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:24 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:24.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:24 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:24.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:24 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:24.869 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- rbd pool init datapool 2024-03-10T03:49:25.110 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T03:49:25.447 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:25 smithi071 ceph-mon[27181]: Detected new or changed devices on smithi071 2024-03-10T03:49:25.447 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:25 smithi071 ceph-mon[27181]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-10T03:49:25.447 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:25 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:25.447 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:25 smithi071 ceph-mon[27181]: osdmap e61: 8 total, 8 up, 8 in 2024-03-10T03:49:25.447 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:25 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/3428248225' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-10T03:49:25.447 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:25 smithi071 ceph-mon[27181]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-10T03:49:25.447 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:25 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:25.447 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:25 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:25.448 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:25 smithi071 ceph-mon[27181]: Saving service rgw.foo.a spec with placement smithi006=foo.a;count:1 2024-03-10T03:49:25.448 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:25 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:25.448 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:25 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:25.448 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:25 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:25.448 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:25 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:25.448 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:25 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:25.448 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:25 smithi071 ceph-mon[27181]: Checking dashboard <-> RGW credentials 2024-03-10T03:49:25.448 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:25 smithi071 ceph-mon[27181]: pgmap v198: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-10T03:49:25.519 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[25256]: Detected new or changed devices on smithi071 2024-03-10T03:49:25.519 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[25256]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-10T03:49:25.519 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:25.519 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[25256]: osdmap e61: 8 total, 8 up, 8 in 2024-03-10T03:49:25.519 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/3428248225' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-10T03:49:25.519 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[25256]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-10T03:49:25.519 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:25.520 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:25.520 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[25256]: Saving service rgw.foo.a spec with placement smithi006=foo.a;count:1 2024-03-10T03:49:25.520 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:25.520 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:25.520 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:25.520 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:25.520 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:25.520 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[25256]: Checking dashboard <-> RGW credentials 2024-03-10T03:49:25.520 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[25256]: pgmap v198: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-10T03:49:25.520 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[31842]: Detected new or changed devices on smithi071 2024-03-10T03:49:25.520 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[31842]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-10T03:49:25.520 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:25.520 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[31842]: osdmap e61: 8 total, 8 up, 8 in 2024-03-10T03:49:25.521 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/3428248225' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-10T03:49:25.521 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[31842]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-10T03:49:25.521 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:25.521 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:25.521 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[31842]: Saving service rgw.foo.a spec with placement smithi006=foo.a;count:1 2024-03-10T03:49:25.521 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:25.521 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:25.521 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:25.521 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:25.521 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:25.521 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[31842]: Checking dashboard <-> RGW credentials 2024-03-10T03:49:25.521 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:25 smithi006 ceph-mon[31842]: pgmap v198: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-10T03:49:26.466 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:26 smithi006 ceph-mon[25256]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-10T03:49:26.466 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:26 smithi006 ceph-mon[25256]: osdmap e62: 8 total, 8 up, 8 in 2024-03-10T03:49:26.466 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:26 smithi006 ceph-mon[25256]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-10T03:49:26.467 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:26 smithi006 ceph-mon[31842]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-10T03:49:26.467 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:26 smithi006 ceph-mon[31842]: osdmap e62: 8 total, 8 up, 8 in 2024-03-10T03:49:26.467 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:26 smithi006 ceph-mon[31842]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-10T03:49:26.521 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:26 smithi071 ceph-mon[27181]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-10T03:49:26.521 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:26 smithi071 ceph-mon[27181]: osdmap e62: 8 total, 8 up, 8 in 2024-03-10T03:49:26.521 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:26 smithi071 ceph-mon[27181]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-10T03:49:28.056 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:27 smithi071 ceph-mon[27181]: osdmap e63: 8 total, 8 up, 8 in 2024-03-10T03:49:28.056 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:27 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/3872279152' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-10T03:49:28.056 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:27 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/967436965' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-10T03:49:28.056 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:27 smithi071 ceph-mon[27181]: from='client.? 172.21.15.71:0/3396285736' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-10T03:49:28.056 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:27 smithi071 ceph-mon[27181]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-10T03:49:28.057 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:27 smithi071 ceph-mon[27181]: pgmap v201: 68 pgs: 7 creating+peering, 53 unknown, 8 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 341 B/s wr, 1 op/s 2024-03-10T03:49:28.057 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:27 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/3872279152' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-10T03:49:28.057 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:27 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/967436965' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-10T03:49:28.057 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:27 smithi071 ceph-mon[27181]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-10T03:49:28.057 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:27 smithi071 ceph-mon[27181]: osdmap e64: 8 total, 8 up, 8 in 2024-03-10T03:49:28.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:27 smithi006 ceph-mon[25256]: osdmap e63: 8 total, 8 up, 8 in 2024-03-10T03:49:28.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:27 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/3872279152' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-10T03:49:28.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:27 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/967436965' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-10T03:49:28.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:27 smithi006 ceph-mon[25256]: from='client.? 172.21.15.71:0/3396285736' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-10T03:49:28.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:27 smithi006 ceph-mon[25256]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-10T03:49:28.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:27 smithi006 ceph-mon[25256]: pgmap v201: 68 pgs: 7 creating+peering, 53 unknown, 8 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 341 B/s wr, 1 op/s 2024-03-10T03:49:28.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:27 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/3872279152' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-10T03:49:28.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:27 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/967436965' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-10T03:49:28.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:27 smithi006 ceph-mon[25256]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-10T03:49:28.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:27 smithi006 ceph-mon[25256]: osdmap e64: 8 total, 8 up, 8 in 2024-03-10T03:49:28.071 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:27 smithi006 ceph-mon[31842]: osdmap e63: 8 total, 8 up, 8 in 2024-03-10T03:49:28.071 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:27 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/3872279152' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-10T03:49:28.071 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:27 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/967436965' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-10T03:49:28.071 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:27 smithi006 ceph-mon[31842]: from='client.? 172.21.15.71:0/3396285736' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-10T03:49:28.071 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:27 smithi006 ceph-mon[31842]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-10T03:49:28.071 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:27 smithi006 ceph-mon[31842]: pgmap v201: 68 pgs: 7 creating+peering, 53 unknown, 8 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 341 B/s wr, 1 op/s 2024-03-10T03:49:28.071 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:27 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/3872279152' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-10T03:49:28.071 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:27 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/967436965' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-10T03:49:28.071 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:27 smithi006 ceph-mon[31842]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-10T03:49:28.071 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:27 smithi006 ceph-mon[31842]: osdmap e64: 8 total, 8 up, 8 in 2024-03-10T03:49:29.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:28 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:29.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:28 smithi071 ceph-mon[27181]: osdmap e65: 8 total, 8 up, 8 in 2024-03-10T03:49:29.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:28 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/967436965' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-10T03:49:29.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:28 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/3872279152' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-10T03:49:29.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:28 smithi071 ceph-mon[27181]: pgmap v204: 100 pgs: 18 creating+peering, 43 unknown, 39 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-03-10T03:49:29.336 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:28 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:29.336 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:28 smithi006 ceph-mon[31842]: osdmap e65: 8 total, 8 up, 8 in 2024-03-10T03:49:29.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:28 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/967436965' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-10T03:49:29.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:28 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/3872279152' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-10T03:49:29.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:28 smithi006 ceph-mon[31842]: pgmap v204: 100 pgs: 18 creating+peering, 43 unknown, 39 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-03-10T03:49:29.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:28 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:29.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:28 smithi006 ceph-mon[25256]: osdmap e65: 8 total, 8 up, 8 in 2024-03-10T03:49:29.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:28 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/967436965' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-10T03:49:29.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:28 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/3872279152' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-10T03:49:29.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:28 smithi006 ceph-mon[25256]: pgmap v204: 100 pgs: 18 creating+peering, 43 unknown, 39 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 767 B/s wr, 1 op/s 2024-03-10T03:49:29.926 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.71 --placement '1;smithi071=iscsi.a' 2024-03-10T03:49:30.166 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T03:49:30.282 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:30 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:30.282 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:30 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/967436965' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-10T03:49:30.282 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:30 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/3872279152' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-10T03:49:30.282 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:30 smithi071 ceph-mon[27181]: osdmap e66: 8 total, 8 up, 8 in 2024-03-10T03:49:30.282 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:30 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:30.282 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:30 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:30.283 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:30 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:30.336 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:30 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:30.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:30 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/967436965' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-10T03:49:30.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:30 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/3872279152' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-10T03:49:30.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:30 smithi006 ceph-mon[31842]: osdmap e66: 8 total, 8 up, 8 in 2024-03-10T03:49:30.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:30 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:30.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:30 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:30.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:30 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:30.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:30 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:30.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:30 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/967436965' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-10T03:49:30.338 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:30 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/3872279152' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-10T03:49:30.338 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:30 smithi006 ceph-mon[25256]: osdmap e66: 8 total, 8 up, 8 in 2024-03-10T03:49:30.338 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:30 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:30.338 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:30 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:30.338 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:30 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:31.466 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:31 smithi071 ceph-mon[27181]: osdmap e67: 8 total, 8 up, 8 in 2024-03-10T03:49:31.466 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:31 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/3872279152' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-10T03:49:31.466 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:31 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/967436965' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-10T03:49:31.466 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:31 smithi071 ceph-mon[27181]: pgmap v207: 132 pgs: 16 creating+peering, 55 unknown, 61 active+clean; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-10T03:49:31.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:31 smithi006 ceph-mon[25256]: osdmap e67: 8 total, 8 up, 8 in 2024-03-10T03:49:31.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:31 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/3872279152' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-10T03:49:31.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:31 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/967436965' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-10T03:49:31.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:31 smithi006 ceph-mon[25256]: pgmap v207: 132 pgs: 16 creating+peering, 55 unknown, 61 active+clean; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-10T03:49:31.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:31 smithi006 ceph-mon[31842]: osdmap e67: 8 total, 8 up, 8 in 2024-03-10T03:49:31.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:31 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/3872279152' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-10T03:49:31.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:31 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/967436965' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-10T03:49:31.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:31 smithi006 ceph-mon[31842]: pgmap v207: 132 pgs: 16 creating+peering, 55 unknown, 61 active+clean; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-10T03:49:31.920 INFO:teuthology.orchestra.run.smithi071.stdout:Scheduled iscsi.datapool update... 2024-03-10T03:49:32.488 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:32 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/3872279152' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-10T03:49:32.488 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:32 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/967436965' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-10T03:49:32.489 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:32 smithi071 ceph-mon[27181]: osdmap e68: 8 total, 8 up, 8 in 2024-03-10T03:49:32.489 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:32 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/3872279152' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-10T03:49:32.489 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:32 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/967436965' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-10T03:49:32.489 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:32 smithi071 ceph-mon[27181]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.71", "placement": "1;smithi071=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:49:32.489 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:32 smithi071 ceph-mon[27181]: Saving service iscsi.datapool spec with placement smithi071=iscsi.a;count:1 2024-03-10T03:49:32.489 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:32 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:32.518 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-10T03:49:32.518 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-10T03:49:32.518 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-10T03:49:32.549 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:32 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/3872279152' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-10T03:49:32.549 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:32 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/967436965' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-10T03:49:32.549 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:32 smithi006 ceph-mon[25256]: osdmap e68: 8 total, 8 up, 8 in 2024-03-10T03:49:32.549 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:32 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/3872279152' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-10T03:49:32.549 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:32 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/967436965' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-10T03:49:32.549 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:32 smithi006 ceph-mon[25256]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.71", "placement": "1;smithi071=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:49:32.549 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:32 smithi006 ceph-mon[25256]: Saving service iscsi.datapool spec with placement smithi071=iscsi.a;count:1 2024-03-10T03:49:32.550 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:32 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:32.550 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:32 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/3872279152' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-10T03:49:32.550 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:32 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/967436965' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-10T03:49:32.550 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:32 smithi006 ceph-mon[31842]: osdmap e68: 8 total, 8 up, 8 in 2024-03-10T03:49:32.550 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:32 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/3872279152' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-10T03:49:32.550 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:32 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/967436965' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-10T03:49:32.550 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:32 smithi006 ceph-mon[31842]: from='client.24376 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.71", "placement": "1;smithi071=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:49:32.550 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:32 smithi006 ceph-mon[31842]: Saving service iscsi.datapool spec with placement smithi071=iscsi.a;count:1 2024-03-10T03:49:32.550 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:32 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:32.554 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-10T03:49:32.554 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-10T03:49:32.586 DEBUG:teuthology.orchestra.run.smithi071:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@iscsi.iscsi.a.service 2024-03-10T03:49:32.628 INFO:tasks.cephadm:Adding prometheus.a on smithi071 2024-03-10T03:49:32.628 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph orch apply prometheus '1;smithi071=a' 2024-03-10T03:49:32.911 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T03:49:33.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:33 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/3872279152' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-10T03:49:33.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:33 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/967436965' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-10T03:49:33.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:33 smithi071 ceph-mon[27181]: osdmap e69: 8 total, 8 up, 8 in 2024-03-10T03:49:33.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:33 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:33.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:33 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:33.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:33 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:33.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:33 smithi071 ceph-mon[27181]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-10T03:49:33.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:33 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-10T03:49:33.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:33 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-10T03:49:33.342 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:33 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:33.342 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:33 smithi071 ceph-mon[27181]: Deploying daemon iscsi.iscsi.a on smithi071 2024-03-10T03:49:33.342 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:33 smithi071 ceph-mon[27181]: pgmap v210: 132 pgs: 10 creating+peering, 6 unknown, 116 active+clean; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 5.0 KiB/s wr, 155 op/s 2024-03-10T03:49:33.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:33 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/3872279152' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-10T03:49:33.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:33 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/967436965' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-10T03:49:33.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:33 smithi006 ceph-mon[25256]: osdmap e69: 8 total, 8 up, 8 in 2024-03-10T03:49:33.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:33 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:33.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:33 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:33.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:33 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:33.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:33 smithi006 ceph-mon[25256]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-10T03:49:33.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:33 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-10T03:49:33.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:33 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-10T03:49:33.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:33 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:33.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:33 smithi006 ceph-mon[25256]: Deploying daemon iscsi.iscsi.a on smithi071 2024-03-10T03:49:33.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:33 smithi006 ceph-mon[25256]: pgmap v210: 132 pgs: 10 creating+peering, 6 unknown, 116 active+clean; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 5.0 KiB/s wr, 155 op/s 2024-03-10T03:49:33.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:33 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/3872279152' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-10T03:49:33.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:33 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/967436965' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-10T03:49:33.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:33 smithi006 ceph-mon[31842]: osdmap e69: 8 total, 8 up, 8 in 2024-03-10T03:49:33.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:33 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:33.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:33 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:33.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:33 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:33.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:33 smithi006 ceph-mon[31842]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-10T03:49:33.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:33 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-10T03:49:33.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:33 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-10T03:49:33.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:33 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:33.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:33 smithi006 ceph-mon[31842]: Deploying daemon iscsi.iscsi.a on smithi071 2024-03-10T03:49:33.589 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:33 smithi006 ceph-mon[31842]: pgmap v210: 132 pgs: 10 creating+peering, 6 unknown, 116 active+clean; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 5.0 KiB/s wr, 155 op/s 2024-03-10T03:49:34.330 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:34 smithi071 ceph-mon[27181]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-10T03:49:34.330 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:34 smithi071 ceph-mon[27181]: Cluster is now healthy 2024-03-10T03:49:34.489 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:34 smithi006 ceph-mon[25256]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-10T03:49:34.489 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:34 smithi006 ceph-mon[25256]: Cluster is now healthy 2024-03-10T03:49:34.490 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:34 smithi006 ceph-mon[31842]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-10T03:49:34.490 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:34 smithi006 ceph-mon[31842]: Cluster is now healthy 2024-03-10T03:49:34.826 INFO:teuthology.orchestra.run.smithi071.stdout:Scheduled prometheus update... 2024-03-10T03:49:35.449 DEBUG:teuthology.orchestra.run.smithi071:prometheus.a> sudo journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@prometheus.a.service 2024-03-10T03:49:35.452 INFO:tasks.cephadm:Adding node-exporter.a on smithi006 2024-03-10T03:49:35.452 INFO:tasks.cephadm:Adding node-exporter.b on smithi071 2024-03-10T03:49:35.452 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph orch apply node-exporter '2;smithi006=a;smithi071=b' 2024-03-10T03:49:35.959 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T03:49:36.077 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:35 smithi071 ceph-mon[27181]: pgmap v211: 132 pgs: 7 creating+peering, 6 unknown, 119 active+clean; 580 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 4.1 KiB/s wr, 139 op/s 2024-03-10T03:49:36.077 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:35 smithi071 ceph-mon[27181]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi071=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:49:36.077 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:35 smithi071 ceph-mon[27181]: Saving service prometheus spec with placement smithi071=a;count:1 2024-03-10T03:49:36.077 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:35 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:36.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:35 smithi006 ceph-mon[25256]: pgmap v211: 132 pgs: 7 creating+peering, 6 unknown, 119 active+clean; 580 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 4.1 KiB/s wr, 139 op/s 2024-03-10T03:49:36.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:35 smithi006 ceph-mon[25256]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi071=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:49:36.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:35 smithi006 ceph-mon[25256]: Saving service prometheus spec with placement smithi071=a;count:1 2024-03-10T03:49:36.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:35 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:36.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:35 smithi006 ceph-mon[31842]: pgmap v211: 132 pgs: 7 creating+peering, 6 unknown, 119 active+clean; 580 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 4.1 KiB/s wr, 139 op/s 2024-03-10T03:49:36.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:35 smithi006 ceph-mon[31842]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi071=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:49:36.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:35 smithi006 ceph-mon[31842]: Saving service prometheus spec with placement smithi071=a;count:1 2024-03-10T03:49:36.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:35 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:37.972 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:37 smithi071 ceph-mon[27181]: pgmap v212: 132 pgs: 132 active+clean; 581 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 4.1 KiB/s wr, 161 op/s 2024-03-10T03:49:37.972 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:37 smithi071 ceph-mon[27181]: osdmap e70: 8 total, 8 up, 8 in 2024-03-10T03:49:38.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:37 smithi006 ceph-mon[25256]: pgmap v212: 132 pgs: 132 active+clean; 581 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 4.1 KiB/s wr, 161 op/s 2024-03-10T03:49:38.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:37 smithi006 ceph-mon[25256]: osdmap e70: 8 total, 8 up, 8 in 2024-03-10T03:49:38.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:37 smithi006 ceph-mon[31842]: pgmap v212: 132 pgs: 132 active+clean; 581 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 4.1 KiB/s wr, 161 op/s 2024-03-10T03:49:38.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:37 smithi006 ceph-mon[31842]: osdmap e70: 8 total, 8 up, 8 in 2024-03-10T03:49:38.428 INFO:teuthology.orchestra.run.smithi071.stdout:Scheduled node-exporter update... 2024-03-10T03:49:38.591 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:49:38 smithi071 podman[47051]: 2024-03-10 03:49:38.530043863 +0000 UTC m=+0.339107261 container start ce51f431d58c2c08ee8cea646575007a5201092d9c58f1071a9b2f51ac37cdd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-10T03:49:39.023 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:49:38 smithi071 bash[47051]: ce51f431d58c2c08ee8cea646575007a5201092d9c58f1071a9b2f51ac37cdd8 2024-03-10T03:49:39.023 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:49:38 smithi071 systemd[1]: Started Ceph iscsi.iscsi.a for 414ba1e0-de90-11ee-95c7-87774f69a715. 2024-03-10T03:49:39.023 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:49:38 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a[47087]: debug Started the configuration object watcher 2024-03-10T03:49:39.023 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:49:38 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a[47087]: debug Checking for config object changes every 1s 2024-03-10T03:49:39.024 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:49:38 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a[47087]: debug Processing osd blocklist entries for this node 2024-03-10T03:49:39.340 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a[47087]: debug Reading the configuration object to update local LIO configuration 2024-03-10T03:49:39.341 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a[47087]: debug Configuration does not have an entry for this host(smithi071.front.sepia.ceph.com) - nothing to define to LIO 2024-03-10T03:49:39.341 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a[47087]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-10T03:49:39.365 DEBUG:teuthology.orchestra.run.smithi006:node-exporter.a> sudo journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@node-exporter.a.service 2024-03-10T03:49:39.367 DEBUG:teuthology.orchestra.run.smithi071:node-exporter.b> sudo journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@node-exporter.b.service 2024-03-10T03:49:39.369 INFO:tasks.cephadm:Adding alertmanager.a on smithi006 2024-03-10T03:49:39.370 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph orch apply alertmanager '1;smithi006=a' 2024-03-10T03:49:39.664 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi006=a;smithi071=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:49:39.665 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: Saving service node-exporter spec with placement smithi006=a;smithi071=b;count:2 2024-03-10T03:49:39.665 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.665 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.665 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.665 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.665 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-10T03:49:39.665 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.665 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 5.0 KiB/s wr, 213 op/s 2024-03-10T03:49:39.665 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:39.665 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:39.665 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:39.665 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.666 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T03:49:39.666 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-10T03:49:39.666 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-10T03:49:39.666 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: Adding iSCSI gateway http://:@172.21.15.71:5000 to Dashboard 2024-03-10T03:49:39.666 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-10T03:49:39.666 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-10T03:49:39.666 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.666 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi071"}]: dispatch 2024-03-10T03:49:39.666 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.666 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:39.666 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:39.666 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:39.666 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: from='client.? 172.21.15.71:0/1818530522' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-10T03:49:39.666 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:39 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi006=a;smithi071=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:49:39.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: Saving service node-exporter spec with placement smithi006=a;smithi071=b;count:2 2024-03-10T03:49:39.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-10T03:49:39.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 5.0 KiB/s wr, 213 op/s 2024-03-10T03:49:39.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:39.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:39.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:39.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T03:49:39.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-10T03:49:39.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-10T03:49:39.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: Adding iSCSI gateway http://:@172.21.15.71:5000 to Dashboard 2024-03-10T03:49:39.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-10T03:49:39.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-10T03:49:39.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi071"}]: dispatch 2024-03-10T03:49:39.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:39.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:39.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:39.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: from='client.? 172.21.15.71:0/1818530522' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-10T03:49:39.838 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi006=a;smithi071=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:49:39.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: Saving service node-exporter spec with placement smithi006=a;smithi071=b;count:2 2024-03-10T03:49:39.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-10T03:49:39.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 5.0 KiB/s wr, 213 op/s 2024-03-10T03:49:39.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:39.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:39.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:39.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T03:49:39.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-10T03:49:39.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-10T03:49:39.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: Adding iSCSI gateway http://:@172.21.15.71:5000 to Dashboard 2024-03-10T03:49:39.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-10T03:49:39.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-10T03:49:39.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi071"}]: dispatch 2024-03-10T03:49:39.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:39.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:39.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:39.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:39.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: from='client.? 172.21.15.71:0/1818530522' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-10T03:49:39.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:39 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:41.023 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:40 smithi071 ceph-mon[27181]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi071"}]: dispatch 2024-03-10T03:49:41.023 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:40 smithi071 ceph-mon[27181]: Checking dashboard <-> RGW credentials 2024-03-10T03:49:41.024 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:40 smithi071 ceph-mon[27181]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T03:49:41.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:40 smithi006 ceph-mon[25256]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi071"}]: dispatch 2024-03-10T03:49:41.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:40 smithi006 ceph-mon[25256]: Checking dashboard <-> RGW credentials 2024-03-10T03:49:41.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:40 smithi006 ceph-mon[25256]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T03:49:41.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:40 smithi006 ceph-mon[31842]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi071"}]: dispatch 2024-03-10T03:49:41.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:40 smithi006 ceph-mon[31842]: Checking dashboard <-> RGW credentials 2024-03-10T03:49:41.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:40 smithi006 ceph-mon[31842]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T03:49:41.486 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T03:49:41.926 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:41 smithi071 ceph-mon[27181]: mgrmap e16: y(active, since 5m), standbys: x 2024-03-10T03:49:41.926 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:41 smithi071 ceph-mon[27181]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 4.3 KiB/s wr, 190 op/s 2024-03-10T03:49:42.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:41 smithi006 ceph-mon[25256]: mgrmap e16: y(active, since 5m), standbys: x 2024-03-10T03:49:42.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:41 smithi006 ceph-mon[25256]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 4.3 KiB/s wr, 190 op/s 2024-03-10T03:49:42.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:41 smithi006 ceph-mon[31842]: mgrmap e16: y(active, since 5m), standbys: x 2024-03-10T03:49:42.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:41 smithi006 ceph-mon[31842]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 4.3 KiB/s wr, 190 op/s 2024-03-10T03:49:43.932 INFO:teuthology.orchestra.run.smithi071.stdout:Scheduled alertmanager update... 2024-03-10T03:49:44.157 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:43 smithi071 ceph-mon[27181]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 2.2 KiB/s wr, 102 op/s 2024-03-10T03:49:44.157 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:43 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:44.157 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:43 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:44.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:43 smithi006 ceph-mon[25256]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 2.2 KiB/s wr, 102 op/s 2024-03-10T03:49:44.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:44.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:43 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:44.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:43 smithi006 ceph-mon[31842]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 2.2 KiB/s wr, 102 op/s 2024-03-10T03:49:44.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:43 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:44.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:43 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:45.183 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:44 smithi071 ceph-mon[27181]: from='client.24469 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi006=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:49:45.183 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:44 smithi071 ceph-mon[27181]: Saving service alertmanager spec with placement smithi006=a;count:1 2024-03-10T03:49:45.183 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:44 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:45.183 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:44 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:45.183 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:44 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:45.183 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:44 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:45.184 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:44 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:45.184 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:44 smithi071 ceph-mon[27181]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T03:49:45.252 DEBUG:teuthology.orchestra.run.smithi006:alertmanager.a> sudo journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@alertmanager.a.service 2024-03-10T03:49:45.254 INFO:tasks.cephadm:Adding grafana.a on smithi071 2024-03-10T03:49:45.255 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph orch apply grafana '1;smithi071=a' 2024-03-10T03:49:45.280 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:44 smithi006 ceph-mon[25256]: from='client.24469 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi006=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:49:45.280 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:44 smithi006 ceph-mon[25256]: Saving service alertmanager spec with placement smithi006=a;count:1 2024-03-10T03:49:45.280 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:44 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:45.280 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:44 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:45.280 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:44 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:45.280 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:44 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:45.280 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:44 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:45.280 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:44 smithi006 ceph-mon[25256]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T03:49:45.280 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:44 smithi006 ceph-mon[31842]: from='client.24469 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi006=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:49:45.281 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:44 smithi006 ceph-mon[31842]: Saving service alertmanager spec with placement smithi006=a;count:1 2024-03-10T03:49:45.281 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:44 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:45.281 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:44 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:45.281 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:44 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:45.281 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:44 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:45.281 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:44 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:45.281 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:44 smithi006 ceph-mon[31842]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-10T03:49:46.200 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T03:49:46.201 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:45 smithi006 ceph-mon[25256]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 1.9 KiB/s wr, 87 op/s 2024-03-10T03:49:46.201 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:45 smithi006 ceph-mon[31842]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 1.9 KiB/s wr, 87 op/s 2024-03-10T03:49:46.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:45 smithi071 ceph-mon[27181]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 1.9 KiB/s wr, 87 op/s 2024-03-10T03:49:47.460 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:47 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:47.460 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:47 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:47.460 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:47 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:47.460 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:47 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:47.461 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:47 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:47.461 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:47 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:47.461 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:47 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:47.461 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:47 smithi006 ceph-mon[31842]: Deploying daemon prometheus.a on smithi071 2024-03-10T03:49:47.461 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:47 smithi006 ceph-mon[31842]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 1.2 KiB/s wr, 58 op/s 2024-03-10T03:49:47.461 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:47 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:47.461 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:47 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:47.461 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:47 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:47.461 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:47 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:47.461 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:47 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:47.461 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:47 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:47.462 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:47 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:47.462 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:47 smithi006 ceph-mon[25256]: Deploying daemon prometheus.a on smithi071 2024-03-10T03:49:47.462 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:47 smithi006 ceph-mon[25256]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 1.2 KiB/s wr, 58 op/s 2024-03-10T03:49:47.572 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:47 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:47.572 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:47 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:47.572 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:47 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:49:47.572 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:47 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:47.572 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:47 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:49:47.573 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:47 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:49:47.573 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:47 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:47.573 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:47 smithi071 ceph-mon[27181]: Deploying daemon prometheus.a on smithi071 2024-03-10T03:49:47.573 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:47 smithi071 ceph-mon[27181]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 1.2 KiB/s wr, 58 op/s 2024-03-10T03:49:47.989 INFO:teuthology.orchestra.run.smithi071.stdout:Scheduled grafana update... 2024-03-10T03:49:48.592 DEBUG:teuthology.orchestra.run.smithi071:grafana.a> sudo journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@grafana.a.service 2024-03-10T03:49:48.595 INFO:tasks.cephadm:Setting up client nodes... 2024-03-10T03:49:48.595 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-10T03:49:48.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:49:48 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a[47087]: debug there is no tcmu-runner data available 2024-03-10T03:49:49.304 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:48 smithi006 ceph-mon[25256]: from='client.24475 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi071=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:49:49.304 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:48 smithi006 ceph-mon[25256]: Saving service grafana spec with placement smithi071=a;count:1 2024-03-10T03:49:49.304 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:48 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:49.304 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:48 smithi006 ceph-mon[25256]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 90 B/s wr, 5 op/s 2024-03-10T03:49:49.304 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:48 smithi006 ceph-mon[25256]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:49:49.304 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:48 smithi006 ceph-mon[31842]: from='client.24475 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi071=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:49:49.304 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:48 smithi006 ceph-mon[31842]: Saving service grafana spec with placement smithi071=a;count:1 2024-03-10T03:49:49.305 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:48 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:49.305 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:48 smithi006 ceph-mon[31842]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 90 B/s wr, 5 op/s 2024-03-10T03:49:49.305 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:48 smithi006 ceph-mon[31842]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:49:49.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:48 smithi071 ceph-mon[27181]: from='client.24475 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi071=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:49:49.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:48 smithi071 ceph-mon[27181]: Saving service grafana spec with placement smithi071=a;count:1 2024-03-10T03:49:49.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:48 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:49.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:48 smithi071 ceph-mon[27181]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 90 B/s wr, 5 op/s 2024-03-10T03:49:49.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:48 smithi071 ceph-mon[27181]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:49:49.383 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:49:50.836 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:50 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:50.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:50 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:50.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:50 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:50.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:50 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:50.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:50 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:50.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:50 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:50.840 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:50 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:50.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:50 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:50.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:50 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:49:51.344 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-10T03:49:51.344 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQDfLe1ly9QvFBAA9AM4TNFMOxfU3GGorSKtIw== 2024-03-10T03:49:51.729 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:51 smithi006 ceph-mon[31842]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-10T03:49:51.729 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:51 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/1371808490' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-10T03:49:51.729 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:51 smithi006 ceph-mon[31842]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-10T03:49:51.729 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:51 smithi006 ceph-mon[31842]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-10T03:49:51.729 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:51 smithi006 ceph-mon[25256]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-10T03:49:51.729 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:51 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/1371808490' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-10T03:49:51.729 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:51 smithi006 ceph-mon[25256]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-10T03:49:51.730 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:51 smithi006 ceph-mon[25256]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-10T03:49:51.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:51 smithi071 ceph-mon[27181]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-10T03:49:51.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:51 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/1371808490' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-10T03:49:51.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:51 smithi071 ceph-mon[27181]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-10T03:49:51.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:51 smithi071 ceph-mon[27181]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-10T03:49:52.038 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-10T03:49:52.038 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-10T03:49:52.038 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-10T03:49:52.083 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-10T03:49:53.837 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.b/config 2024-03-10T03:49:54.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:53 smithi006 ceph-mon[25256]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-10T03:49:54.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:53 smithi006 ceph-mon[31842]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-10T03:49:54.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:53 smithi071 ceph-mon[27181]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-10T03:49:56.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:55 smithi006 ceph-mon[25256]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:49:56.086 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:55 smithi006 ceph-mon[31842]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:49:56.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:55 smithi071 ceph-mon[27181]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:49:58.085 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:57 smithi071 ceph-mon[27181]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:49:58.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:57 smithi006 ceph-mon[25256]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:49:58.086 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:57 smithi006 ceph-mon[31842]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:49:59.091 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:49:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a[47087]: debug there is no tcmu-runner data available 2024-03-10T03:49:59.201 INFO:teuthology.orchestra.run.smithi071.stdout:[client.1] 2024-03-10T03:49:59.201 INFO:teuthology.orchestra.run.smithi071.stdout: key = AQDnLe1l8CaWCxAAHbXUtmgCeSyMkpqBThuBDA== 2024-03-10T03:49:59.824 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-10T03:49:59.824 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-10T03:49:59.824 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-10T03:49:59.869 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-10T03:49:59.869 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-10T03:49:59.869 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph mgr dump --format=json 2024-03-10T03:49:59.921 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:59 smithi071 ceph-mon[27181]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:49:59.922 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:59 smithi071 ceph-mon[27181]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:49:59.922 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:59 smithi071 ceph-mon[27181]: from='client.? 172.21.15.71:0/718564775' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-10T03:49:59.922 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:59 smithi071 ceph-mon[27181]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-10T03:49:59.922 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:49:59 smithi071 ceph-mon[27181]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-10T03:50:00.122 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:50:00.179 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:50:00 smithi071 podman[48831]: 2024-03-10 03:50:00.15434441 +0000 UTC m=+1.177081100 container start 008f9b15c15b3c75925af7fe13a177d461d4784d9619a250bc009158233ddf6d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-10T03:50:00.179 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:50:00.178Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-10T03:50:00.179 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:50:00.178Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-10T03:50:00.179 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:50:00.178Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi071 (none))" 2024-03-10T03:50:00.179 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:50:00.178Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-10T03:50:00.180 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:50:00.178Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-10T03:50:00.243 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:59 smithi006 ceph-mon[31842]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:00.243 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:59 smithi006 ceph-mon[31842]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:50:00.244 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:59 smithi006 ceph-mon[31842]: from='client.? 172.21.15.71:0/718564775' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-10T03:50:00.244 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:59 smithi006 ceph-mon[31842]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-10T03:50:00.244 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:49:59 smithi006 ceph-mon[31842]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-10T03:50:00.244 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:59 smithi006 ceph-mon[25256]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:00.244 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:59 smithi006 ceph-mon[25256]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:50:00.244 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:59 smithi006 ceph-mon[25256]: from='client.? 172.21.15.71:0/718564775' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-10T03:50:00.244 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:59 smithi006 ceph-mon[25256]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-10T03:50:00.244 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:49:59 smithi006 ceph-mon[25256]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-10T03:50:00.444 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:50:00.179Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-10T03:50:00.445 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:50:00.180Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-10T03:50:00.445 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:50:00.181Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-10T03:50:00.445 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:50:00.182Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-10T03:50:00.445 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:50:00.183Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-10T03:50:00.445 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:50:00.183Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.574µs 2024-03-10T03:50:00.445 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:50:00.183Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-10T03:50:00.445 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:50:00.183Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-10T03:50:00.445 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:50:00.183Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.143µs wal_replay_duration=196.523µs wbl_replay_duration=86ns total_replay_duration=232.521µs 2024-03-10T03:50:00.445 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:50:00.184Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-10T03:50:00.445 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:50:00.184Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-10T03:50:00.446 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:50:00.184Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-10T03:50:00.446 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:50:00.204Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.976922ms db_storage=678ns remote_storage=1.114µs web_handler=260ns query_engine=434ns scrape=441.07µs scrape_sd=79.589µs notify=678ns notify_sd=1.865µs rules=19.256389ms tracing=6.344µs 2024-03-10T03:50:00.446 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:50:00.204Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-10T03:50:00.446 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:50:00.204Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-10T03:50:00.446 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:50:00 smithi071 bash[48831]: 008f9b15c15b3c75925af7fe13a177d461d4784d9619a250bc009158233ddf6d 2024-03-10T03:50:00.446 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:50:00 smithi071 systemd[1]: Started Ceph prometheus.a for 414ba1e0-de90-11ee-95c7-87774f69a715. 2024-03-10T03:50:01.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-mon[27181]: overall HEALTH_OK 2024-03-10T03:50:01.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:50:01.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:50:01.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:50:01.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:00 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-10T03:50:01.253 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:00 smithi006 ceph-mon[25256]: overall HEALTH_OK 2024-03-10T03:50:01.253 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:00 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:50:01.253 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:00 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:50:01.253 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:00 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:50:01.253 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:00 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-10T03:50:01.253 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:00 smithi006 ceph-mon[31842]: overall HEALTH_OK 2024-03-10T03:50:01.253 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:00 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:50:01.253 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:00 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:50:01.254 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:00 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' 2024-03-10T03:50:01.254 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:00 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-10T03:50:01.536 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:01 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: ignoring --setuser ceph since I am not root 2024-03-10T03:50:01.536 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:01 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: ignoring --setgroup ceph since I am not root 2024-03-10T03:50:01.537 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:01 smithi071 ceph-mgr[29448]: -- 172.21.15.71:0/1193966971 <== mon.1 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e39997a5a0 con 0x55e39994ec00 2024-03-10T03:50:01.551 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:01 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: ignoring --setuser ceph since I am not root 2024-03-10T03:50:01.552 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:01 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: ignoring --setgroup ceph since I am not root 2024-03-10T03:50:02.056 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:50:02.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:01 smithi071 ceph-mon[27181]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:50:02.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:01 smithi071 ceph-mon[27181]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-10T03:50:02.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:01 smithi071 ceph-mon[27181]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-10T03:50:02.091 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:01 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:01.892+0000 7f66c2d5e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-10T03:50:02.091 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:01 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:01.990+0000 7f66c2d5e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-10T03:50:02.094 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:01 smithi006 ceph-mon[25256]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:50:02.094 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:01 smithi006 ceph-mon[25256]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-10T03:50:02.094 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:01 smithi006 ceph-mon[25256]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-10T03:50:02.094 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:01 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:01.946+0000 7faaa99b5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-10T03:50:02.095 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:02 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:02.043+0000 7faaa99b5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-10T03:50:02.095 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:01 smithi006 ceph-mon[31842]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:50:02.095 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:01 smithi006 ceph-mon[31842]: from='mgr.14152 172.21.15.6:0/858398912' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-10T03:50:02.095 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:01 smithi006 ceph-mon[31842]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-10T03:50:02.752 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:02 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:02.657+0000 7f66c2d5e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-10T03:50:02.872 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":1055327650},{"type":"v1","addr":"172.21.15.6:6801","nonce":1055327650}]},"active_addr":"172.21.15.6:6801/1055327650","active_change":"2024-03-10T03:44:32.888945+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24115,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.6:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":1888187587}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":861381203}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":3195685682}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":73381501}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":3555584422}]}]} 2024-03-10T03:50:02.878 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-10T03:50:02.878 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-10T03:50:02.879 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph osd dump --format=json 2024-03-10T03:50:02.990 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:02 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/809863024' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-10T03:50:02.990 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:02 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/809863024' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-10T03:50:02.990 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:02 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:02.729+0000 7faaa99b5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-10T03:50:02.991 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:02 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:02.825+0000 7faaa99b5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-10T03:50:03.073 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:02 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/809863024' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-10T03:50:03.073 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:02 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:02.751+0000 7f66c2d5e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-10T03:50:03.073 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:02 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:02.894+0000 7f66c2d5e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-10T03:50:03.073 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:02 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:02.989+0000 7f66c2d5e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-10T03:50:03.073 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:03 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:03.072+0000 7f66c2d5e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-10T03:50:03.141 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:50:03.270 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:02 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:02.988+0000 7faaa99b5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-10T03:50:03.270 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:03 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:03.093+0000 7faaa99b5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-10T03:50:03.270 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:03 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:03.195+0000 7faaa99b5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-10T03:50:03.568 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:03 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:03.473+0000 7faaa99b5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-10T03:50:03.697 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:03 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:03.369+0000 7f66c2d5e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-10T03:50:03.697 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:03 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:03.501+0000 7f66c2d5e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-10T03:50:03.697 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:03 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:03.594+0000 7f66c2d5e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-10T03:50:03.836 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:03 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:03.588+0000 7faaa99b5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-10T03:50:03.836 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:03 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:03.677+0000 7faaa99b5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-10T03:50:03.836 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:03 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:03.775+0000 7faaa99b5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-10T03:50:04.002 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:03 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:03.696+0000 7f66c2d5e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-10T03:50:04.002 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:03 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:03.883+0000 7f66c2d5e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-10T03:50:04.280 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:03 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:03.956+0000 7faaa99b5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-10T03:50:05.009 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:50:05.009 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":70,"fsid":"414ba1e0-de90-11ee-95c7-87774f69a715","created":"2024-03-10T03:43:29.585078+0000","modified":"2024-03-10T03:49:37.482893+0000","last_up_change":"2024-03-10T03:49:08.927831+0000","last_in_change":"2024-03-10T03:48:50.519693+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-10T03:47:01.225820+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-10T03:49:23.800585+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":66,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-10T03:49:24.185836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-10T03:49:25.529778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-10T03:49:27.477933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-10T03:49:29.538557+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"58176ab1-73a2-4d74-8202-a581cd7e934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":726018156},{"type":"v1","addr":"172.21.15.6:6803","nonce":726018156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":726018156},{"type":"v1","addr":"172.21.15.6:6805","nonce":726018156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":726018156},{"type":"v1","addr":"172.21.15.6:6809","nonce":726018156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":726018156},{"type":"v1","addr":"172.21.15.6:6807","nonce":726018156}]},"public_addr":"172.21.15.6:6803/726018156","cluster_addr":"172.21.15.6:6805/726018156","heartbeat_back_addr":"172.21.15.6:6809/726018156","heartbeat_front_addr":"172.21.15.6:6807/726018156","state":["exists","up"]},{"osd":1,"uuid":"b4eee59a-8e2b-4d51-8f0e-feb6600222b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":399164807},{"type":"v1","addr":"172.21.15.6:6811","nonce":399164807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":399164807},{"type":"v1","addr":"172.21.15.6:6813","nonce":399164807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":399164807},{"type":"v1","addr":"172.21.15.6:6817","nonce":399164807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":399164807},{"type":"v1","addr":"172.21.15.6:6815","nonce":399164807}]},"public_addr":"172.21.15.6:6811/399164807","cluster_addr":"172.21.15.6:6813/399164807","heartbeat_back_addr":"172.21.15.6:6817/399164807","heartbeat_front_addr":"172.21.15.6:6815/399164807","state":["exists","up"]},{"osd":2,"uuid":"eadadcfe-1185-4ea7-9eee-cb2c346b70b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3078148920},{"type":"v1","addr":"172.21.15.6:6819","nonce":3078148920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3078148920},{"type":"v1","addr":"172.21.15.6:6821","nonce":3078148920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3078148920},{"type":"v1","addr":"172.21.15.6:6825","nonce":3078148920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3078148920},{"type":"v1","addr":"172.21.15.6:6823","nonce":3078148920}]},"public_addr":"172.21.15.6:6819/3078148920","cluster_addr":"172.21.15.6:6821/3078148920","heartbeat_back_addr":"172.21.15.6:6825/3078148920","heartbeat_front_addr":"172.21.15.6:6823/3078148920","state":["exists","up"]},{"osd":3,"uuid":"ecb7b3ca-493f-4837-9772-146a4f323ba4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":388589465},{"type":"v1","addr":"172.21.15.6:6827","nonce":388589465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":388589465},{"type":"v1","addr":"172.21.15.6:6829","nonce":388589465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":388589465},{"type":"v1","addr":"172.21.15.6:6833","nonce":388589465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":388589465},{"type":"v1","addr":"172.21.15.6:6831","nonce":388589465}]},"public_addr":"172.21.15.6:6827/388589465","cluster_addr":"172.21.15.6:6829/388589465","heartbeat_back_addr":"172.21.15.6:6833/388589465","heartbeat_front_addr":"172.21.15.6:6831/388589465","state":["exists","up"]},{"osd":4,"uuid":"5cdd2e4a-82c4-422a-a716-1ac8527991a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":150275056},{"type":"v1","addr":"172.21.15.71:6801","nonce":150275056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":150275056},{"type":"v1","addr":"172.21.15.71:6803","nonce":150275056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":150275056},{"type":"v1","addr":"172.21.15.71:6807","nonce":150275056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":150275056},{"type":"v1","addr":"172.21.15.71:6805","nonce":150275056}]},"public_addr":"172.21.15.71:6801/150275056","cluster_addr":"172.21.15.71:6803/150275056","heartbeat_back_addr":"172.21.15.71:6807/150275056","heartbeat_front_addr":"172.21.15.71:6805/150275056","state":["exists","up"]},{"osd":5,"uuid":"c1f68af1-e235-4363-8e31-19ea142661fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":3012063760},{"type":"v1","addr":"172.21.15.71:6809","nonce":3012063760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3012063760},{"type":"v1","addr":"172.21.15.71:6811","nonce":3012063760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":3012063760},{"type":"v1","addr":"172.21.15.71:6815","nonce":3012063760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":3012063760},{"type":"v1","addr":"172.21.15.71:6813","nonce":3012063760}]},"public_addr":"172.21.15.71:6809/3012063760","cluster_addr":"172.21.15.71:6811/3012063760","heartbeat_back_addr":"172.21.15.71:6815/3012063760","heartbeat_front_addr":"172.21.15.71:6813/3012063760","state":["exists","up"]},{"osd":6,"uuid":"8e07f4f7-b672-4a77-abe5-3a440eff844f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2652028816},{"type":"v1","addr":"172.21.15.71:6817","nonce":2652028816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2652028816},{"type":"v1","addr":"172.21.15.71:6819","nonce":2652028816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2652028816},{"type":"v1","addr":"172.21.15.71:6823","nonce":2652028816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2652028816},{"type":"v1","addr":"172.21.15.71:6821","nonce":2652028816}]},"public_addr":"172.21.15.71:6817/2652028816","cluster_addr":"172.21.15.71:6819/2652028816","heartbeat_back_addr":"172.21.15.71:6823/2652028816","heartbeat_front_addr":"172.21.15.71:6821/2652028816","state":["exists","up"]},{"osd":7,"uuid":"aaeda1f0-6c59-44a2-b394-ebacf817248c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3011457307},{"type":"v1","addr":"172.21.15.71:6825","nonce":3011457307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":3011457307},{"type":"v1","addr":"172.21.15.71:6827","nonce":3011457307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":3011457307},{"type":"v1","addr":"172.21.15.71:6831","nonce":3011457307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":3011457307},{"type":"v1","addr":"172.21.15.71:6829","nonce":3011457307}]},"public_addr":"172.21.15.71:6825/3011457307","cluster_addr":"172.21.15.71:6827/3011457307","heartbeat_back_addr":"172.21.15.71:6831/3011457307","heartbeat_front_addr":"172.21.15.71:6829/3011457307","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T03:46:03.779380+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T03:46:30.107772+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T03:46:57.905469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T03:47:25.581989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T03:47:49.843184+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T03:48:14.623236+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T03:48:41.677581+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T03:49:07.269732+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:0/1227130243":"2024-03-11T03:44:32.888833+0000","172.21.15.6:6801/194403410":"2024-03-11T03:44:32.888833+0000","172.21.15.6:6800/194403410":"2024-03-11T03:44:32.888833+0000","172.21.15.6:0/3339850563":"2024-03-11T03:44:32.888833+0000","172.21.15.6:0/446417578":"2024-03-11T03:44:05.341856+0000","172.21.15.6:0/437052514":"2024-03-11T03:44:05.341856+0000","172.21.15.6:6800/3433712073":"2024-03-11T03:44:05.341856+0000","172.21.15.6:6801/3433712073":"2024-03-11T03:44:05.341856+0000","172.21.15.6:0/1672617939":"2024-03-11T03:44:05.341856+0000","172.21.15.6:0/2959683900":"2024-03-11T03:44:32.888833+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-10T03:50:05.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:05 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/3182724427' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-10T03:50:05.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:05 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/3182724427' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-10T03:50:05.337 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:04 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:04.955+0000 7faaa99b5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-10T03:50:05.337 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:05.098+0000 7faaa99b5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-10T03:50:05.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:05 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/3182724427' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-10T03:50:05.340 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:04 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:04.882+0000 7f66c2d5e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-10T03:50:05.341 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:05 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:05.018+0000 7f66c2d5e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-10T03:50:05.645 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:05 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:05.358+0000 7f66c2d5e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-10T03:50:05.645 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:05 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:05.448+0000 7f66c2d5e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-10T03:50:05.645 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:05 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:05.644+0000 7f66c2d5e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-10T03:50:05.735 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:05.439+0000 7faaa99b5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-10T03:50:05.736 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:05.532+0000 7faaa99b5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-10T03:50:05.781 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-10T03:50:05.781 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph osd dump --format=json 2024-03-10T03:50:05.938 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:05 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:05.836+0000 7f66c2d5e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-10T03:50:06.032 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:50:06.086 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:05.734+0000 7faaa99b5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-10T03:50:06.086 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:05.941+0000 7faaa99b5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-10T03:50:06.340 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:06 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:06.131+0000 7f66c2d5e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-10T03:50:06.416 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:06 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:06.249+0000 7faaa99b5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-10T03:50:06.636 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:06 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:06.397+0000 7f66c2d5e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-10T03:50:06.636 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:06 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:06.635+0000 7f66c2d5e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-10T03:50:06.743 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:06 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:06.519+0000 7faaa99b5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-10T03:50:07.011 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:06 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:06.766+0000 7faaa99b5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-10T03:50:07.673 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:07 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:07.608+0000 7faaa99b5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-10T03:50:07.840 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:07 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:07.431+0000 7f66c2d5e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-10T03:50:08.309 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:07 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:07.998+0000 7f66c2d5e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-10T03:50:08.310 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:08 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:08.095+0000 7f66c2d5e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-10T03:50:08.310 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:08 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:08.308+0000 7f66c2d5e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-10T03:50:08.382 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:08 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:08.203+0000 7faaa99b5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-10T03:50:08.382 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:08 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:08.305+0000 7faaa99b5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-10T03:50:08.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:08 smithi071 ceph-mon[27181]: Standby manager daemon x restarted 2024-03-10T03:50:08.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:08 smithi071 ceph-mon[27181]: Standby manager daemon x started 2024-03-10T03:50:08.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:08 smithi071 ceph-mon[27181]: from='mgr.? 172.21.15.71:0/2650442322' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-10T03:50:08.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:08 smithi071 ceph-mon[27181]: from='mgr.? 172.21.15.71:0/2650442322' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-10T03:50:08.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:08 smithi071 ceph-mon[27181]: from='mgr.? 172.21.15.71:0/2650442322' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-10T03:50:08.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:08 smithi071 ceph-mon[27181]: from='mgr.? 172.21.15.71:0/2650442322' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-10T03:50:08.591 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:08 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: [10/Mar/2024:03:50:08] ENGINE Bus STARTING 2024-03-10T03:50:08.591 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:08 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: CherryPy Checker: 2024-03-10T03:50:08.591 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:08 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: The Application mounted at '' has an empty config. 2024-03-10T03:50:08.591 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:08 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: 2024-03-10T03:50:08.592 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:08 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: [10/Mar/2024:03:50:08] ENGINE Serving on http://:::9283 2024-03-10T03:50:08.592 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:50:08 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x[29444]: [10/Mar/2024:03:50:08] ENGINE Bus STARTED 2024-03-10T03:50:08.659 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:08 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:08.537+0000 7faaa99b5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-10T03:50:08.660 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:08 smithi006 ceph-mon[25256]: Standby manager daemon x restarted 2024-03-10T03:50:08.660 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:08 smithi006 ceph-mon[25256]: Standby manager daemon x started 2024-03-10T03:50:08.660 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:08 smithi006 ceph-mon[25256]: from='mgr.? 172.21.15.71:0/2650442322' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-10T03:50:08.660 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:08 smithi006 ceph-mon[25256]: from='mgr.? 172.21.15.71:0/2650442322' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-10T03:50:08.660 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:08 smithi006 ceph-mon[25256]: from='mgr.? 172.21.15.71:0/2650442322' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-10T03:50:08.660 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:08 smithi006 ceph-mon[25256]: from='mgr.? 172.21.15.71:0/2650442322' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-10T03:50:08.660 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:08 smithi006 ceph-mon[31842]: Standby manager daemon x restarted 2024-03-10T03:50:08.661 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:08 smithi006 ceph-mon[31842]: Standby manager daemon x started 2024-03-10T03:50:08.661 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:08 smithi006 ceph-mon[31842]: from='mgr.? 172.21.15.71:0/2650442322' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-10T03:50:08.661 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:08 smithi006 ceph-mon[31842]: from='mgr.? 172.21.15.71:0/2650442322' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-10T03:50:08.661 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:08 smithi006 ceph-mon[31842]: from='mgr.? 172.21.15.71:0/2650442322' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-10T03:50:08.661 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:08 smithi006 ceph-mon[31842]: from='mgr.? 172.21.15.71:0/2650442322' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-10T03:50:09.006 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:08 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:50:08] ENGINE Bus STARTING 2024-03-10T03:50:09.006 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:08 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: CherryPy Checker: 2024-03-10T03:50:09.006 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:08 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: The Application mounted at '' has an empty config. 2024-03-10T03:50:09.006 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:08 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:50:09.006 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:08 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:50:08] ENGINE Serving on http://:::9283 2024-03-10T03:50:09.006 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:08 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:50:08] ENGINE Bus STARTED 2024-03-10T03:50:09.404 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:50:09.404 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":71,"fsid":"414ba1e0-de90-11ee-95c7-87774f69a715","created":"2024-03-10T03:43:29.585078+0000","modified":"2024-03-10T03:50:08.552035+0000","last_up_change":"2024-03-10T03:49:08.927831+0000","last_in_change":"2024-03-10T03:48:50.519693+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-10T03:47:01.225820+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-10T03:49:23.800585+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":66,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-10T03:49:24.185836+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-10T03:49:25.529778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-10T03:49:27.477933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-10T03:49:29.538557+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"58176ab1-73a2-4d74-8202-a581cd7e934e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":726018156},{"type":"v1","addr":"172.21.15.6:6803","nonce":726018156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":726018156},{"type":"v1","addr":"172.21.15.6:6805","nonce":726018156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":726018156},{"type":"v1","addr":"172.21.15.6:6809","nonce":726018156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":726018156},{"type":"v1","addr":"172.21.15.6:6807","nonce":726018156}]},"public_addr":"172.21.15.6:6803/726018156","cluster_addr":"172.21.15.6:6805/726018156","heartbeat_back_addr":"172.21.15.6:6809/726018156","heartbeat_front_addr":"172.21.15.6:6807/726018156","state":["exists","up"]},{"osd":1,"uuid":"b4eee59a-8e2b-4d51-8f0e-feb6600222b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":399164807},{"type":"v1","addr":"172.21.15.6:6811","nonce":399164807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":399164807},{"type":"v1","addr":"172.21.15.6:6813","nonce":399164807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":399164807},{"type":"v1","addr":"172.21.15.6:6817","nonce":399164807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":399164807},{"type":"v1","addr":"172.21.15.6:6815","nonce":399164807}]},"public_addr":"172.21.15.6:6811/399164807","cluster_addr":"172.21.15.6:6813/399164807","heartbeat_back_addr":"172.21.15.6:6817/399164807","heartbeat_front_addr":"172.21.15.6:6815/399164807","state":["exists","up"]},{"osd":2,"uuid":"eadadcfe-1185-4ea7-9eee-cb2c346b70b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6818","nonce":3078148920},{"type":"v1","addr":"172.21.15.6:6819","nonce":3078148920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6820","nonce":3078148920},{"type":"v1","addr":"172.21.15.6:6821","nonce":3078148920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6824","nonce":3078148920},{"type":"v1","addr":"172.21.15.6:6825","nonce":3078148920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6822","nonce":3078148920},{"type":"v1","addr":"172.21.15.6:6823","nonce":3078148920}]},"public_addr":"172.21.15.6:6819/3078148920","cluster_addr":"172.21.15.6:6821/3078148920","heartbeat_back_addr":"172.21.15.6:6825/3078148920","heartbeat_front_addr":"172.21.15.6:6823/3078148920","state":["exists","up"]},{"osd":3,"uuid":"ecb7b3ca-493f-4837-9772-146a4f323ba4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6826","nonce":388589465},{"type":"v1","addr":"172.21.15.6:6827","nonce":388589465}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6828","nonce":388589465},{"type":"v1","addr":"172.21.15.6:6829","nonce":388589465}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6832","nonce":388589465},{"type":"v1","addr":"172.21.15.6:6833","nonce":388589465}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6830","nonce":388589465},{"type":"v1","addr":"172.21.15.6:6831","nonce":388589465}]},"public_addr":"172.21.15.6:6827/388589465","cluster_addr":"172.21.15.6:6829/388589465","heartbeat_back_addr":"172.21.15.6:6833/388589465","heartbeat_front_addr":"172.21.15.6:6831/388589465","state":["exists","up"]},{"osd":4,"uuid":"5cdd2e4a-82c4-422a-a716-1ac8527991a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":150275056},{"type":"v1","addr":"172.21.15.71:6801","nonce":150275056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":150275056},{"type":"v1","addr":"172.21.15.71:6803","nonce":150275056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":150275056},{"type":"v1","addr":"172.21.15.71:6807","nonce":150275056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":150275056},{"type":"v1","addr":"172.21.15.71:6805","nonce":150275056}]},"public_addr":"172.21.15.71:6801/150275056","cluster_addr":"172.21.15.71:6803/150275056","heartbeat_back_addr":"172.21.15.71:6807/150275056","heartbeat_front_addr":"172.21.15.71:6805/150275056","state":["exists","up"]},{"osd":5,"uuid":"c1f68af1-e235-4363-8e31-19ea142661fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":3012063760},{"type":"v1","addr":"172.21.15.71:6809","nonce":3012063760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":3012063760},{"type":"v1","addr":"172.21.15.71:6811","nonce":3012063760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":3012063760},{"type":"v1","addr":"172.21.15.71:6815","nonce":3012063760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":3012063760},{"type":"v1","addr":"172.21.15.71:6813","nonce":3012063760}]},"public_addr":"172.21.15.71:6809/3012063760","cluster_addr":"172.21.15.71:6811/3012063760","heartbeat_back_addr":"172.21.15.71:6815/3012063760","heartbeat_front_addr":"172.21.15.71:6813/3012063760","state":["exists","up"]},{"osd":6,"uuid":"8e07f4f7-b672-4a77-abe5-3a440eff844f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":2652028816},{"type":"v1","addr":"172.21.15.71:6817","nonce":2652028816}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":2652028816},{"type":"v1","addr":"172.21.15.71:6819","nonce":2652028816}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":2652028816},{"type":"v1","addr":"172.21.15.71:6823","nonce":2652028816}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":2652028816},{"type":"v1","addr":"172.21.15.71:6821","nonce":2652028816}]},"public_addr":"172.21.15.71:6817/2652028816","cluster_addr":"172.21.15.71:6819/2652028816","heartbeat_back_addr":"172.21.15.71:6823/2652028816","heartbeat_front_addr":"172.21.15.71:6821/2652028816","state":["exists","up"]},{"osd":7,"uuid":"aaeda1f0-6c59-44a2-b394-ebacf817248c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":3011457307},{"type":"v1","addr":"172.21.15.71:6825","nonce":3011457307}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":3011457307},{"type":"v1","addr":"172.21.15.71:6827","nonce":3011457307}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":3011457307},{"type":"v1","addr":"172.21.15.71:6831","nonce":3011457307}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":3011457307},{"type":"v1","addr":"172.21.15.71:6829","nonce":3011457307}]},"public_addr":"172.21.15.71:6825/3011457307","cluster_addr":"172.21.15.71:6827/3011457307","heartbeat_back_addr":"172.21.15.71:6831/3011457307","heartbeat_front_addr":"172.21.15.71:6829/3011457307","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T03:46:03.779380+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T03:46:30.107772+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T03:46:57.905469+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T03:47:25.581989+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T03:47:49.843184+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T03:48:14.623236+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T03:48:41.677581+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-10T03:49:07.269732+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:0/1888187587":"2024-03-11T03:50:08.551943+0000","172.21.15.6:0/2959683900":"2024-03-11T03:44:32.888833+0000","172.21.15.6:0/1672617939":"2024-03-11T03:44:05.341856+0000","172.21.15.6:0/861381203":"2024-03-11T03:50:08.551943+0000","172.21.15.6:6801/3433712073":"2024-03-11T03:44:05.341856+0000","172.21.15.6:6801/1055327650":"2024-03-11T03:50:08.551943+0000","172.21.15.6:0/3555584422":"2024-03-11T03:50:08.551943+0000","172.21.15.6:6800/3433712073":"2024-03-11T03:44:05.341856+0000","172.21.15.6:0/437052514":"2024-03-11T03:44:05.341856+0000","172.21.15.6:0/3339850563":"2024-03-11T03:44:32.888833+0000","172.21.15.6:6800/1055327650":"2024-03-11T03:50:08.551943+0000","172.21.15.6:6800/194403410":"2024-03-11T03:44:32.888833+0000","172.21.15.6:6801/194403410":"2024-03-11T03:44:32.888833+0000","172.21.15.6:0/1227130243":"2024-03-11T03:44:32.888833+0000","172.21.15.6:0/446417578":"2024-03-11T03:44:05.341856+0000","172.21.15.6:0/3195685682":"2024-03-11T03:50:08.551943+0000","172.21.15.6:0/73381501":"2024-03-11T03:50:08.551943+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-10T03:50:09.663 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-10T03:50:09.663 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: Active manager daemon y restarted 2024-03-10T03:50:09.663 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: Activating manager daemon y 2024-03-10T03:50:09.663 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: osdmap e71: 8 total, 8 up, 8 in 2024-03-10T03:50:09.663 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: mgrmap e19: y(active, starting, since 0.00870903s), standbys: x 2024-03-10T03:50:09.663 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T03:50:09.664 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:50:09.664 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:50:09.664 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-10T03:50:09.664 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-10T03:50:09.664 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T03:50:09.664 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T03:50:09.664 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T03:50:09.664 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T03:50:09.664 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T03:50:09.664 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T03:50:09.664 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T03:50:09.664 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T03:50:09.664 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-10T03:50:09.664 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-10T03:50:09.664 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-10T03:50:09.665 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: Manager daemon y is now available 2024-03-10T03:50:09.665 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:09.665 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:50:09.665 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:50:09.665 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi006", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:50:09.665 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi006", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:50:09.665 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi071", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:50:09.665 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi006", "name": "osd_memory_target"}]': finished 2024-03-10T03:50:09.665 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-10T03:50:09.665 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: Active manager daemon y restarted 2024-03-10T03:50:09.665 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: Activating manager daemon y 2024-03-10T03:50:09.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: osdmap e71: 8 total, 8 up, 8 in 2024-03-10T03:50:09.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: mgrmap e19: y(active, starting, since 0.00870903s), standbys: x 2024-03-10T03:50:09.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T03:50:09.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:50:09.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:50:09.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-10T03:50:09.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-10T03:50:09.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T03:50:09.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T03:50:09.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T03:50:09.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T03:50:09.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T03:50:09.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T03:50:09.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T03:50:09.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T03:50:09.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-10T03:50:09.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-10T03:50:09.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-10T03:50:09.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: Manager daemon y is now available 2024-03-10T03:50:09.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:09.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:50:09.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:50:09.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi006", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:50:09.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi006", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:50:09.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi071", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:50:09.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi006", "name": "osd_memory_target"}]': finished 2024-03-10T03:50:09.669 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi071", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:50:09.669 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi071", "name": "osd_memory_target"}]': finished 2024-03-10T03:50:09.670 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:50:09.670 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:09.670 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:50:09.670 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T03:50:09.670 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-10T03:50:09.670 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-10T03:50:09.670 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-10T03:50:09.670 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-10T03:50:09.670 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi071", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:50:09.670 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi071", "name": "osd_memory_target"}]': finished 2024-03-10T03:50:09.670 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:50:09.670 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:09.671 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:50:09.671 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T03:50:09.671 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-10T03:50:09.671 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-10T03:50:09.671 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-10T03:50:09.671 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:09 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-10T03:50:09.840 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a[47087]: debug there is no tcmu-runner data available 2024-03-10T03:50:09.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-10T03:50:09.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: Active manager daemon y restarted 2024-03-10T03:50:09.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: Activating manager daemon y 2024-03-10T03:50:09.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: osdmap e71: 8 total, 8 up, 8 in 2024-03-10T03:50:09.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: mgrmap e19: y(active, starting, since 0.00870903s), standbys: x 2024-03-10T03:50:09.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-10T03:50:09.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-10T03:50:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-10T03:50:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-10T03:50:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-10T03:50:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-10T03:50:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-10T03:50:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-10T03:50:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-10T03:50:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-10T03:50:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-10T03:50:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-10T03:50:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-10T03:50:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-10T03:50:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-10T03:50:09.842 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-10T03:50:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: Manager daemon y is now available 2024-03-10T03:50:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:50:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:50:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi006", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:50:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi006", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:50:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi071", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:50:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi006", "name": "osd_memory_target"}]': finished 2024-03-10T03:50:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi071", "name": "osd_memory_target"}]: dispatch 2024-03-10T03:50:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi071", "name": "osd_memory_target"}]': finished 2024-03-10T03:50:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:50:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:50:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T03:50:09.843 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-10T03:50:09.844 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-10T03:50:09.844 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-10T03:50:09.844 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:09 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-10T03:50:10.425 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-10T03:50:10.426 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-10T03:50:10.426 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-10T03:50:10.427 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-10T03:50:10.427 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-10T03:50:10.427 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-10T03:50:10.428 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-10T03:50:10.428 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-10T03:50:10.704 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:10 smithi071 ceph-mon[27181]: Updating smithi006:/etc/ceph/ceph.conf 2024-03-10T03:50:10.704 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:10 smithi071 ceph-mon[27181]: Updating smithi071:/etc/ceph/ceph.conf 2024-03-10T03:50:10.704 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:10 smithi071 ceph-mon[27181]: [10/Mar/2024:03:50:09] ENGINE Bus STARTING 2024-03-10T03:50:10.704 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:10 smithi071 ceph-mon[27181]: [10/Mar/2024:03:50:09] ENGINE Serving on http://172.21.15.6:8765 2024-03-10T03:50:10.704 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:10 smithi071 ceph-mon[27181]: Updating smithi071:/var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T03:50:10.704 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:10 smithi071 ceph-mon[27181]: Updating smithi006:/var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T03:50:10.704 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:10 smithi071 ceph-mon[27181]: [10/Mar/2024:03:50:09] ENGINE Serving on https://172.21.15.6:7151 2024-03-10T03:50:10.704 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:10 smithi071 ceph-mon[27181]: [10/Mar/2024:03:50:09] ENGINE Bus STARTED 2024-03-10T03:50:10.705 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:10 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/1318859947' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-10T03:50:10.705 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:10 smithi071 ceph-mon[27181]: Updating smithi071:/etc/ceph/ceph.client.admin.keyring 2024-03-10T03:50:10.705 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:10 smithi071 ceph-mon[27181]: Updating smithi006:/etc/ceph/ceph.client.admin.keyring 2024-03-10T03:50:10.705 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:10 smithi071 ceph-mon[27181]: mgrmap e20: y(active, since 1.0161s), standbys: x 2024-03-10T03:50:10.705 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:10 smithi071 ceph-mon[27181]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:50:10.705 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:10 smithi071 ceph-mon[27181]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-10T03:50:10.705 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:10 smithi071 ceph-mon[27181]: Updating smithi071:/var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/config/ceph.client.admin.keyring 2024-03-10T03:50:10.705 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:10 smithi071 ceph-mon[27181]: Updating smithi006:/var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/config/ceph.client.admin.keyring 2024-03-10T03:50:10.705 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:10 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:10.705 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:10 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:10.705 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:10 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:10.705 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:10 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:10.705 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:10 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:10.705 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:10 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:10.705 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:10 smithi071 ceph-mon[27181]: Deploying daemon alertmanager.a on smithi006 2024-03-10T03:50:10.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[31842]: Updating smithi006:/etc/ceph/ceph.conf 2024-03-10T03:50:10.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[31842]: Updating smithi071:/etc/ceph/ceph.conf 2024-03-10T03:50:10.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[31842]: [10/Mar/2024:03:50:09] ENGINE Bus STARTING 2024-03-10T03:50:10.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[31842]: [10/Mar/2024:03:50:09] ENGINE Serving on http://172.21.15.6:8765 2024-03-10T03:50:10.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[31842]: Updating smithi071:/var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T03:50:10.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[31842]: Updating smithi006:/var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T03:50:10.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[31842]: [10/Mar/2024:03:50:09] ENGINE Serving on https://172.21.15.6:7151 2024-03-10T03:50:10.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[31842]: [10/Mar/2024:03:50:09] ENGINE Bus STARTED 2024-03-10T03:50:10.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/1318859947' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-10T03:50:10.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[31842]: Updating smithi071:/etc/ceph/ceph.client.admin.keyring 2024-03-10T03:50:10.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[31842]: Updating smithi006:/etc/ceph/ceph.client.admin.keyring 2024-03-10T03:50:10.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[31842]: mgrmap e20: y(active, since 1.0161s), standbys: x 2024-03-10T03:50:10.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[31842]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:50:10.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[31842]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-10T03:50:10.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[31842]: Updating smithi071:/var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/config/ceph.client.admin.keyring 2024-03-10T03:50:10.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[31842]: Updating smithi006:/var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/config/ceph.client.admin.keyring 2024-03-10T03:50:10.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:10.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:10.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:10.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:10.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:10.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:10.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[31842]: Deploying daemon alertmanager.a on smithi006 2024-03-10T03:50:10.839 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[25256]: Updating smithi006:/etc/ceph/ceph.conf 2024-03-10T03:50:10.839 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[25256]: Updating smithi071:/etc/ceph/ceph.conf 2024-03-10T03:50:10.839 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[25256]: [10/Mar/2024:03:50:09] ENGINE Bus STARTING 2024-03-10T03:50:10.839 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[25256]: [10/Mar/2024:03:50:09] ENGINE Serving on http://172.21.15.6:8765 2024-03-10T03:50:10.839 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[25256]: Updating smithi071:/var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T03:50:10.839 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[25256]: Updating smithi006:/var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/config/ceph.conf 2024-03-10T03:50:10.839 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[25256]: [10/Mar/2024:03:50:09] ENGINE Serving on https://172.21.15.6:7151 2024-03-10T03:50:10.839 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[25256]: [10/Mar/2024:03:50:09] ENGINE Bus STARTED 2024-03-10T03:50:10.839 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/1318859947' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-10T03:50:10.839 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[25256]: Updating smithi071:/etc/ceph/ceph.client.admin.keyring 2024-03-10T03:50:10.839 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[25256]: Updating smithi006:/etc/ceph/ceph.client.admin.keyring 2024-03-10T03:50:10.839 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[25256]: mgrmap e20: y(active, since 1.0161s), standbys: x 2024-03-10T03:50:10.839 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[25256]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:50:10.839 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[25256]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-10T03:50:10.840 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[25256]: Updating smithi071:/var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/config/ceph.client.admin.keyring 2024-03-10T03:50:10.840 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[25256]: Updating smithi006:/var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/config/ceph.client.admin.keyring 2024-03-10T03:50:10.840 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:10.840 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:10.840 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:10.840 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:10.840 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:10.840 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:10.840 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:10 smithi006 ceph-mon[25256]: Deploying daemon alertmanager.a on smithi006 2024-03-10T03:50:10.996 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:50:11.000 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:50:11.028 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:50:11.078 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:50:11.078 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:50:11.083 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:50:11.590 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:11 smithi071 ceph-mon[27181]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-10T03:50:11.731 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:11 smithi006 ceph-mon[25256]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-10T03:50:11.731 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:11 smithi006 ceph-mon[31842]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-10T03:50:12.830 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:12 smithi006 ceph-mon[31842]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-10T03:50:12.831 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:12 smithi006 ceph-mon[25256]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-10T03:50:12.840 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:12 smithi071 ceph-mon[27181]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-10T03:50:13.086 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:12 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: ::ffff:172.21.15.71 - - [10/Mar/2024:03:50:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-10T03:50:14.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:13 smithi006 ceph-mon[25256]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-10T03:50:14.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:13 smithi006 ceph-mon[25256]: mgrmap e22: y(active, since 5s), standbys: x 2024-03-10T03:50:14.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:13 smithi006 ceph-mon[31842]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-10T03:50:14.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:13 smithi006 ceph-mon[31842]: mgrmap e22: y(active, since 5s), standbys: x 2024-03-10T03:50:14.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:13 smithi071 ceph-mon[27181]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-10T03:50:14.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:13 smithi071 ceph-mon[27181]: mgrmap e22: y(active, since 5s), standbys: x 2024-03-10T03:50:15.885 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:15 smithi071 ceph-mon[27181]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-10T03:50:15.944 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:50:16.015 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:15 smithi006 ceph-mon[25256]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-10T03:50:16.015 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:15 smithi006 ceph-mon[31842]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-03-10T03:50:17.749 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:17 smithi006 ceph-mon[25256]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-10T03:50:17.749 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:17 smithi006 ceph-mon[31842]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-10T03:50:18.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:17 smithi071 ceph-mon[27181]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-10T03:50:19.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:19 smithi006 ceph-mon[25256]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-10T03:50:19.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:19 smithi006 ceph-mon[31842]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-10T03:50:19.840 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:19 smithi071 ceph-mon[27181]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-10T03:50:19.841 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:50:19 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a[47087]: debug there is no tcmu-runner data available 2024-03-10T03:50:20.890 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:20 smithi071 ceph-mon[27181]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:50:20.904 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:20 smithi006 ceph-mon[25256]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:50:20.905 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:20 smithi006 ceph-mon[31842]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:50:21.920 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:21 smithi006 ceph-mon[25256]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-10T03:50:21.920 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:21 smithi006 ceph-mon[31842]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-10T03:50:22.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:21 smithi071 ceph-mon[27181]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-10T03:50:22.229 INFO:teuthology.orchestra.run.smithi006.stdout:133143986208 2024-03-10T03:50:22.229 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-10T03:50:23.020 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:22 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: ::ffff:172.21.15.71 - - [10/Mar/2024:03:50:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-10T03:50:23.819 INFO:teuthology.orchestra.run.smithi006.stdout:111669149733 2024-03-10T03:50:23.820 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-10T03:50:23.840 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:23 smithi071 ceph-mon[27181]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-10T03:50:23.992 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:23 smithi006 ceph-mon[25256]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-10T03:50:23.993 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:23 smithi006 ceph-mon[31842]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-10T03:50:24.600 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:50:24.619 INFO:teuthology.orchestra.run.smithi006.stdout:167503724571 2024-03-10T03:50:24.619 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-10T03:50:24.977 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:24 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T03:50:24.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:24 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T03:50:25.002 INFO:teuthology.orchestra.run.smithi006.stdout:77309411371 2024-03-10T03:50:25.002 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-10T03:50:25.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:24 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T03:50:25.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:25 smithi006 ceph-mon[25256]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-10T03:50:25.836 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:25 smithi006 ceph-mon[31842]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-10T03:50:25.958 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:25 smithi071 ceph-mon[27181]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-10T03:50:26.586 INFO:teuthology.orchestra.run.smithi006.stdout:240518168592 2024-03-10T03:50:26.586 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-10T03:50:28.003 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:27 smithi006 ceph-mon[25256]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-10T03:50:28.003 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:27 smithi006 ceph-mon[31842]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-10T03:50:28.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:27 smithi071 ceph-mon[27181]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-10T03:50:28.578 INFO:teuthology.orchestra.run.smithi006.stdout:55834574896 2024-03-10T03:50:28.579 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-10T03:50:29.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:29 smithi006 ceph-mon[25256]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:29.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:29 smithi006 ceph-mon[31842]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:30.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:29 smithi071 ceph-mon[27181]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:30.091 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:50:29 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a[47087]: debug there is no tcmu-runner data available 2024-03-10T03:50:30.444 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:50:30.444 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:50:30.836 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:30 smithi006 ceph-mon[31842]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:50:30.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:30 smithi006 ceph-mon[25256]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:50:31.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:30 smithi071 ceph-mon[27181]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:50:31.369 INFO:teuthology.orchestra.run.smithi006.stdout:206158430231 2024-03-10T03:50:31.370 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-10T03:50:31.566 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:50:31.566 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:50:31.963 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:31 smithi006 ceph-mon[25256]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:50:31.963 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:31 smithi006 ceph-mon[31842]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:50:32.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:31 smithi071 ceph-mon[27181]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:50:32.600 INFO:teuthology.orchestra.run.smithi006.stdout:34359738424 2024-03-10T03:50:32.600 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-10T03:50:33.086 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:32 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: ::ffff:172.21.15.71 - - [10/Mar/2024:03:50:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-10T03:50:33.928 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:33 smithi006 ceph-mon[25256]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:33.928 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:33 smithi006 ceph-mon[31842]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:34.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:33 smithi071 ceph-mon[27181]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:35.587 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:50:35 smithi006 systemd[1]: Starting Ceph alertmanager.a for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:50:35.866 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:35 smithi006 ceph-mon[25256]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:35.867 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:35 smithi006 ceph-mon[31842]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:36.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:35 smithi071 ceph-mon[27181]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:36.610 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:50:38.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:37 smithi006 ceph-mon[25256]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:50:38.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:37 smithi006 ceph-mon[31842]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:50:38.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:37 smithi071 ceph-mon[27181]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:50:38.532 INFO:teuthology.orchestra.run.smithi006.stdout:111669149736 2024-03-10T03:50:38.730 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:38 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/1988558272' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-10T03:50:38.731 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:38 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/1988558272' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-10T03:50:39.046 INFO:teuthology.orchestra.run.smithi006.stdout:240518168596 2024-03-10T03:50:39.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:38 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/1988558272' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-10T03:50:39.789 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:39 smithi006 ceph-mon[25256]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:39.790 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:39 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T03:50:39.790 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:39 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/2367477269' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-10T03:50:39.790 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:39 smithi006 ceph-mon[31842]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:39.790 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:39 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T03:50:39.790 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:39 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/2367477269' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-10T03:50:40.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:39 smithi071 ceph-mon[27181]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:40.093 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:39 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T03:50:40.093 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:39 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/2367477269' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-10T03:50:40.093 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:50:39 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a[47087]: debug there is no tcmu-runner data available 2024-03-10T03:50:40.332 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:50:40 smithi006 podman[59810]: 2024-03-10 03:50:40.197655729 +0000 UTC m=+0.023157313 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-10T03:50:40.586 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:50:40 smithi006 podman[59810]: 2024-03-10 03:50:40.332071802 +0000 UTC m=+0.157573375 volume create 7e1982548e5c4966fdb8913315094fe094e388d785df41b92d41c9c4e47b2933 2024-03-10T03:50:40.744 INFO:teuthology.orchestra.run.smithi006.stdout:167503724575 2024-03-10T03:50:40.880 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:40 smithi071 ceph-mon[27181]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:50:40.883 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:40 smithi006 ceph-mon[25256]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:50:40.884 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:40 smithi006 ceph-mon[31842]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:50:41.963 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:41 smithi006 ceph-mon[25256]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:50:41.963 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:41 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/2332567541' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-10T03:50:41.964 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:50:41 smithi006 podman[59810]: 2024-03-10 03:50:41.763681326 +0000 UTC m=+1.589182938 container create 7ae68de3709a4c81af7b837523b6826fa99141616d6ec5bc4b476c27317aee81 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-10T03:50:41.964 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:41 smithi006 ceph-mon[31842]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:50:41.964 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:41 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/2332567541' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-10T03:50:42.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:41 smithi071 ceph-mon[27181]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:50:42.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:41 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/2332567541' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-10T03:50:42.897 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:42 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: ::ffff:172.21.15.71 - - [10/Mar/2024:03:50:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-10T03:50:43.020 INFO:teuthology.orchestra.run.smithi006.stdout:133143986212 2024-03-10T03:50:43.629 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168592 got 240518168596 for osd.7 2024-03-10T03:50:43.630 DEBUG:teuthology.parallel:result is None 2024-03-10T03:50:43.942 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:43 smithi006 ceph-mon[25256]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:43.942 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:43 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/1543391790' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-10T03:50:43.943 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:43 smithi006 ceph-mon[31842]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:43.943 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:43 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/1543391790' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-10T03:50:43.943 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:50:43 smithi006 podman[59810]: 2024-03-10 03:50:43.704373329 +0000 UTC m=+3.529874906 container init 7ae68de3709a4c81af7b837523b6826fa99141616d6ec5bc4b476c27317aee81 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-10T03:50:43.943 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:50:43 smithi006 podman[59810]: 2024-03-10 03:50:43.707291538 +0000 UTC m=+3.532793116 container start 7ae68de3709a4c81af7b837523b6826fa99141616d6ec5bc4b476c27317aee81 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-10T03:50:43.943 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:50:43 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[59930]: ts=2024-03-10T03:50:43.719Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-10T03:50:43.944 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:50:43 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[59930]: ts=2024-03-10T03:50:43.719Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-10T03:50:43.944 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:50:43 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[59930]: ts=2024-03-10T03:50:43.720Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.6 port=9094 2024-03-10T03:50:43.944 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:50:43 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[59930]: ts=2024-03-10T03:50:43.721Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-10T03:50:43.944 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:50:43 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[59930]: ts=2024-03-10T03:50:43.745Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-10T03:50:43.944 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:50:43 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[59930]: ts=2024-03-10T03:50:43.745Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-10T03:50:43.944 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:50:43 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[59930]: ts=2024-03-10T03:50:43.748Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-10T03:50:43.944 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:50:43 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[59930]: ts=2024-03-10T03:50:43.748Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-10T03:50:43.944 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:50:43 smithi006 bash[59810]: 7ae68de3709a4c81af7b837523b6826fa99141616d6ec5bc4b476c27317aee81 2024-03-10T03:50:43.944 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:50:43 smithi006 systemd[1]: Started Ceph alertmanager.a for 414ba1e0-de90-11ee-95c7-87774f69a715. 2024-03-10T03:50:44.013 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:43 smithi071 ceph-mon[27181]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:44.014 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:43 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/1543391790' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-10T03:50:44.336 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:43 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:50:43] ENGINE Bus STOPPING 2024-03-10T03:50:44.336 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:43 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:50:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-10T03:50:44.336 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:43 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:50:43] ENGINE Bus STOPPED 2024-03-10T03:50:44.337 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:43 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:50:43] ENGINE Bus STARTING 2024-03-10T03:50:44.337 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:50:44] ENGINE Serving on http://:::9283 2024-03-10T03:50:44.337 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:50:44] ENGINE Bus STARTED 2024-03-10T03:50:44.486 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149733 got 111669149736 for osd.3 2024-03-10T03:50:44.487 DEBUG:teuthology.parallel:result is None 2024-03-10T03:50:44.568 INFO:teuthology.orchestra.run.smithi006.stdout:77309411375 2024-03-10T03:50:45.072 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:45.072 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:45.072 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:45.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:45.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-mon[31842]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-10T03:50:45.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:45.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:45.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-10T03:50:45.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-mon[31842]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-10T03:50:45.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:45.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-mon[31842]: Deploying daemon grafana.a on smithi071 2024-03-10T03:50:45.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/1345857063' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-10T03:50:45.073 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:45.073 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:45.073 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:45.074 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:45.074 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-mon[25256]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-10T03:50:45.074 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:45.074 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:45.074 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-10T03:50:45.074 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-mon[25256]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-10T03:50:45.074 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:45.074 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-mon[25256]: Deploying daemon grafana.a on smithi071 2024-03-10T03:50:45.074 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:44 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/1345857063' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-10T03:50:45.100 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411371 got 77309411375 for osd.2 2024-03-10T03:50:45.100 DEBUG:teuthology.parallel:result is None 2024-03-10T03:50:45.309 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:50:45.309 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:50:45.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:44 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:45.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:44 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:45.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:44 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:45.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:44 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:45.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:44 smithi071 ceph-mon[27181]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-10T03:50:45.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:44 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:45.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:44 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:45.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:44 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-10T03:50:45.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:44 smithi071 ceph-mon[27181]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-10T03:50:45.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:44 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:45.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:44 smithi071 ceph-mon[27181]: Deploying daemon grafana.a on smithi071 2024-03-10T03:50:45.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:44 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/1345857063' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-10T03:50:45.731 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986208 got 133143986212 for osd.4 2024-03-10T03:50:45.731 DEBUG:teuthology.parallel:result is None 2024-03-10T03:50:46.047 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:45 smithi006 ceph-mon[31842]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:46.047 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:50:45 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[59930]: ts=2024-03-10T03:50:45.721Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000121661s 2024-03-10T03:50:46.047 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:45 smithi006 ceph-mon[25256]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:46.076 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724571 got 167503724575 for osd.5 2024-03-10T03:50:46.077 DEBUG:teuthology.parallel:result is None 2024-03-10T03:50:46.078 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:50:46.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:45 smithi071 ceph-mon[27181]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:48.089 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:47 smithi006 ceph-mon[25256]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:50:48.089 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:47 smithi006 ceph-mon[31842]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:50:48.340 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:47 smithi071 ceph-mon[27181]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:50:48.907 INFO:teuthology.orchestra.run.smithi006.stdout:206158430234 2024-03-10T03:50:49.907 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:49 smithi006 ceph-mon[31842]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:49.907 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:49 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:49.907 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:49 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/743048287' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-10T03:50:49.908 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:49 smithi006 ceph-mon[25256]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:49.908 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:49 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:49.908 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:49 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/743048287' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-10T03:50:50.090 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:50:49 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a[47087]: debug there is no tcmu-runner data available 2024-03-10T03:50:50.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:49 smithi071 ceph-mon[27181]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:50.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:49 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:50.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:49 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/743048287' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-10T03:50:50.958 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:50 smithi006 ceph-mon[25256]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:50:50.959 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:50 smithi006 ceph-mon[31842]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:50:51.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:50 smithi071 ceph-mon[27181]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:50:51.099 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430231 got 206158430234 for osd.6 2024-03-10T03:50:51.099 DEBUG:teuthology.parallel:result is None 2024-03-10T03:50:51.945 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:51 smithi006 ceph-mon[25256]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:50:51.945 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:51 smithi006 ceph-mon[31842]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:50:52.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:51 smithi071 ceph-mon[27181]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:50:53.030 INFO:teuthology.orchestra.run.smithi006.stdout:34359738428 2024-03-10T03:50:53.031 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:50:52 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: ::ffff:172.21.15.71 - - [10/Mar/2024:03:50:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-10T03:50:53.878 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738424 got 34359738428 for osd.0 2024-03-10T03:50:53.879 DEBUG:teuthology.parallel:result is None 2024-03-10T03:50:53.956 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:53 smithi006 ceph-mon[31842]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:53.956 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:53 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/650613652' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-10T03:50:53.957 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:50:53 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[59930]: ts=2024-03-10T03:50:53.723Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001260441s 2024-03-10T03:50:53.957 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:53 smithi006 ceph-mon[25256]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:53.957 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:53 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/650613652' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-10T03:50:54.049 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:53 smithi071 ceph-mon[27181]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:50:54.049 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:53 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/650613652' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-10T03:50:54.531 INFO:teuthology.orchestra.run.smithi006.stdout:55834574902 2024-03-10T03:50:54.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:54 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T03:50:54.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:54 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/4127687937' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-10T03:50:54.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:54 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T03:50:54.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:54 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/4127687937' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-10T03:50:54.971 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:54 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T03:50:54.971 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:54 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/4127687937' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-10T03:50:55.822 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:55 smithi006 ceph-mon[25256]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:50:55.822 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:55 smithi006 ceph-mon[31842]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:50:55.889 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574896 got 55834574902 for osd.1 2024-03-10T03:50:55.889 DEBUG:teuthology.parallel:result is None 2024-03-10T03:50:55.889 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-10T03:50:55.889 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph pg dump --format=json 2024-03-10T03:50:55.974 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:55 smithi071 ceph-mon[27181]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:50:56.374 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:50:57.722 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:57 smithi006 ceph-mon[25256]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:50:57.723 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:57 smithi006 ceph-mon[31842]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:50:57.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:57 smithi071 ceph-mon[27181]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:50:58.199 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 podman[51262]: 2024-03-10 03:50:58.117828847 +0000 UTC m=+0.362549286 container start 1c68ee40279e3da4885ebd6774b38dea2803009cf2f157afd3e659d4e9ab3882 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., architecture=x86_64, version=8.5, release=239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, vcs-type=git, name=ubi8, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-03-10T03:50:58.199 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 bash[51262]: 1c68ee40279e3da4885ebd6774b38dea2803009cf2f157afd3e659d4e9ab3882 2024-03-10T03:50:58.450 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 systemd[1]: Started Ceph grafana.a for 414ba1e0-de90-11ee-95c7-87774f69a715. 2024-03-10T03:50:58.450 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=settings t=2024-03-10T03:50:58.269506574Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-10T03:50:58.450 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=settings t=2024-03-10T03:50:58.269692218Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-10T03:50:58.450 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=settings t=2024-03-10T03:50:58.269710926Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-10T03:50:58.450 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=settings t=2024-03-10T03:50:58.269718308Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-10T03:50:58.450 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=settings t=2024-03-10T03:50:58.269724225Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-10T03:50:58.451 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=settings t=2024-03-10T03:50:58.269730061Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-10T03:50:58.451 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=settings t=2024-03-10T03:50:58.269738893Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-10T03:50:58.451 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=settings t=2024-03-10T03:50:58.269744774Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-10T03:50:58.451 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=settings t=2024-03-10T03:50:58.2697505Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-10T03:50:58.451 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=settings t=2024-03-10T03:50:58.269756044Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-10T03:50:58.451 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=settings t=2024-03-10T03:50:58.269761283Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-10T03:50:58.451 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=settings t=2024-03-10T03:50:58.269766634Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-10T03:50:58.451 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=settings t=2024-03-10T03:50:58.269772193Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-10T03:50:58.451 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=settings t=2024-03-10T03:50:58.269777701Z level=info msg="App mode production" 2024-03-10T03:50:58.451 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=sqlstore t=2024-03-10T03:50:58.269827361Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-10T03:50:58.451 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=sqlstore t=2024-03-10T03:50:58.269843131Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-10T03:50:58.452 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.270732298Z level=info msg="Starting DB migrations" 2024-03-10T03:50:58.452 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.270807353Z level=info msg="Executing migration" id="create migration_log table" 2024-03-10T03:50:58.452 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.271339325Z level=info msg="Executing migration" id="create user table" 2024-03-10T03:50:58.452 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.271758507Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-10T03:50:58.452 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.272277752Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-10T03:50:58.452 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.273040337Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-10T03:50:58.452 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.273912323Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-10T03:50:58.452 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.274901359Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-10T03:50:58.452 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.275694489Z level=info msg="Executing migration" id="create user table v2" 2024-03-10T03:50:58.452 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.276116562Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-10T03:50:58.452 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.277099779Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-10T03:50:58.452 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.277963891Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-10T03:50:58.452 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.27866177Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-10T03:50:58.452 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.279477873Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-10T03:50:58.453 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.279928418Z level=info msg="Executing migration" id="Update user table charset" 2024-03-10T03:50:58.453 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.280348415Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-10T03:50:58.453 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.281273754Z level=info msg="Executing migration" id="Add missing user data" 2024-03-10T03:50:58.453 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.281717474Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-10T03:50:58.453 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.282159451Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-10T03:50:58.453 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.283024557Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-10T03:50:58.453 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.284006495Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-10T03:50:58.453 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.285335318Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-10T03:50:58.454 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.286210753Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-10T03:50:58.454 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.287163696Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-10T03:50:58.454 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.29015693Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-10T03:50:58.454 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.29285616Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-10T03:50:58.454 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.295508734Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-10T03:50:58.454 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.298045575Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-10T03:50:58.455 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.301005008Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-10T03:50:58.455 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.303663427Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-10T03:50:58.455 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.306522507Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-10T03:50:58.455 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.309212352Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-10T03:50:58.455 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.309868777Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-10T03:50:58.455 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.312582388Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-10T03:50:58.455 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.315269373Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-10T03:50:58.455 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.318061509Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-10T03:50:58.455 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.318476355Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-10T03:50:58.455 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.321204821Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-10T03:50:58.455 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.32386359Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-10T03:50:58.455 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.326681737Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-10T03:50:58.455 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.329549738Z level=info msg="Executing migration" id="create star table" 2024-03-10T03:50:58.456 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.332254162Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-10T03:50:58.456 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.335034076Z level=info msg="Executing migration" id="create org table v1" 2024-03-10T03:50:58.456 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.337702035Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-10T03:50:58.456 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.340332111Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-10T03:50:58.456 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.343076744Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-10T03:50:58.456 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.345932282Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-10T03:50:58.456 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.348636641Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-10T03:50:58.456 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.351365872Z level=info msg="Executing migration" id="Update org table charset" 2024-03-10T03:50:58.456 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.354150917Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-10T03:50:58.456 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.356738757Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-10T03:50:58.456 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.359333678Z level=info msg="Executing migration" id="create dashboard table" 2024-03-10T03:50:58.456 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.362132496Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-10T03:50:58.456 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.365019807Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-10T03:50:58.456 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.36758038Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-10T03:50:58.457 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.370345198Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-10T03:50:58.457 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.373014969Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-10T03:50:58.457 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.375669781Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-10T03:50:58.457 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.378171764Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-10T03:50:58.457 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.379107171Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-10T03:50:58.457 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.379864239Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-10T03:50:58.457 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.380762123Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-10T03:50:58.457 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.381418115Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-10T03:50:58.457 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.384000254Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-10T03:50:58.457 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.386494685Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-10T03:50:58.457 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.38743262Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-10T03:50:58.457 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.388131111Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-10T03:50:58.457 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.389072235Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-10T03:50:58.458 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.389844421Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-10T03:50:58.458 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.390689146Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-10T03:50:58.458 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.391446025Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-10T03:50:58.458 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.392308448Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-10T03:50:58.458 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.3930605Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-10T03:50:58.458 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.393996451Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-10T03:50:58.458 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.394646619Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-10T03:50:58.458 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.395359738Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-10T03:50:58.458 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.396207324Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-10T03:50:58.458 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.397174182Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-10T03:50:58.458 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.397890009Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-10T03:50:58.458 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.398846111Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-10T03:50:58.458 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.399583815Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-10T03:50:58.459 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.400478764Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-10T03:50:58.459 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.40109927Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-10T03:50:58.459 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.401488628Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-10T03:50:58.459 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.402360383Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-10T03:50:58.459 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.403238347Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-10T03:50:58.459 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.404223242Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-10T03:50:58.459 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.404968237Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-10T03:50:58.459 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.405839365Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-10T03:50:58.459 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.406682819Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-10T03:50:58.459 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.407537088Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-10T03:50:58.459 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.408386325Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-10T03:50:58.459 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.409272957Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-10T03:50:58.459 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.410189764Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-10T03:50:58.459 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.411071299Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-10T03:50:58.460 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.411813863Z level=info msg="Executing migration" id="create data_source table" 2024-03-10T03:50:58.460 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.412693755Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-10T03:50:58.460 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.413478338Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-10T03:50:58.460 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.413904072Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-10T03:50:58.460 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.414772927Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-10T03:50:58.460 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.415730362Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-10T03:50:58.460 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.416808938Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-10T03:50:58.460 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.419647151Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-10T03:50:58.460 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.422470384Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-10T03:50:58.460 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.425445529Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-10T03:50:58.460 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.428050318Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-10T03:50:58.460 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.430598636Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-10T03:50:58.461 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.43354885Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-10T03:50:58.461 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.436592705Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-10T03:50:58.461 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.436990919Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-10T03:50:58.461 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.437575464Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-10T03:50:58.461 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.437882756Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-10T03:50:58.461 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.438255449Z level=info msg="Executing migration" id="Add uid column" 2024-03-10T03:50:58.461 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.440505093Z level=info msg="Executing migration" id="Update uid value" 2024-03-10T03:50:58.461 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.440926038Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-10T03:50:58.461 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.443124402Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-10T03:50:58.461 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.446067189Z level=info msg="Executing migration" id="create api_key table" 2024-03-10T03:50:58.461 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.4491504Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-10T03:50:58.461 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.451864539Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-10T03:50:58.701 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.454691411Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-10T03:50:58.701 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.457431303Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-10T03:50:58.701 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.457830099Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-10T03:50:58.701 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.458235779Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-10T03:50:58.701 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.458630925Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-10T03:50:58.701 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.459642104Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-10T03:50:58.702 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.460099463Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-10T03:50:58.702 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.460504086Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-10T03:50:58.702 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.460905299Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-10T03:50:58.702 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.46133468Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-10T03:50:58.702 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.461665941Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-10T03:50:58.702 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.462000354Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-10T03:50:58.702 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.4622931Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-10T03:50:58.702 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.462798751Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-10T03:50:58.702 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.463339039Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-10T03:50:58.702 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.463649019Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-10T03:50:58.702 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.464198868Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-10T03:50:58.702 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.464734223Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-10T03:50:58.703 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.465138997Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-10T03:50:58.703 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.465506137Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-10T03:50:58.703 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.465877007Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-10T03:50:58.703 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.466308287Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-10T03:50:58.703 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.46671934Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-10T03:50:58.703 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.467128911Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-10T03:50:58.703 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.467431435Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-10T03:50:58.703 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.467692874Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-10T03:50:58.703 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.468258921Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-10T03:50:58.703 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.468839682Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-10T03:50:58.703 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.469161066Z level=info msg="Executing migration" id="create quota table v1" 2024-03-10T03:50:58.703 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.469553806Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-10T03:50:58.703 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.469973384Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-10T03:50:58.703 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.470270208Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-10T03:50:58.704 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.470652298Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-10T03:50:58.704 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.471101811Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-10T03:50:58.704 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.47166724Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-10T03:50:58.704 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.4719549Z level=info msg="Executing migration" id="create session table" 2024-03-10T03:50:58.704 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.472383564Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-10T03:50:58.704 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.472672725Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-10T03:50:58.705 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.472987434Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-10T03:50:58.705 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.475297087Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-10T03:50:58.705 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.475706025Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-10T03:50:58.705 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.475973066Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-10T03:50:58.705 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.476263643Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-10T03:50:58.706 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.476558321Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-10T03:50:58.706 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.476823556Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-10T03:50:58.706 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.477229429Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-10T03:50:58.706 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.47761286Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-10T03:50:58.706 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.478446138Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-10T03:50:58.706 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.478741644Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-10T03:50:58.706 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.479320191Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-10T03:50:58.706 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.47988174Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-10T03:50:58.706 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.480179811Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-10T03:50:58.706 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.480585944Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-10T03:50:58.706 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.481047361Z level=info msg="Executing migration" id="create alert table v1" 2024-03-10T03:50:58.706 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.483802141Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-10T03:50:58.706 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.486641962Z level=info msg="Executing migration" id="add index alert state" 2024-03-10T03:50:58.707 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.489042261Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-10T03:50:58.707 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.489479366Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-10T03:50:58.707 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.4921318Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-10T03:50:58.707 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.494766763Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-10T03:50:58.707 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.49736995Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-10T03:50:58.707 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.500065137Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-10T03:50:58.707 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.502807263Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-10T03:50:58.707 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.505484266Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-10T03:50:58.707 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.505881809Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-10T03:50:58.707 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.508012728Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-10T03:50:58.707 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.510683571Z level=info msg="Executing migration" id="Add column is_default" 2024-03-10T03:50:58.707 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.513406736Z level=info msg="Executing migration" id="Add column frequency" 2024-03-10T03:50:58.707 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.516160266Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-10T03:50:58.707 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.517041995Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-10T03:50:58.708 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.519623065Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-10T03:50:58.708 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.522426542Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-10T03:50:58.708 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.522738838Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-10T03:50:58.708 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.523007921Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-10T03:50:58.708 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.525092135Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-10T03:50:58.708 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.527797032Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-10T03:50:58.708 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.528620838Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-10T03:50:58.708 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.529553629Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-10T03:50:58.708 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.530059141Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-10T03:50:58.708 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.530852314Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-10T03:50:58.708 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.531665774Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-10T03:50:58.708 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.532523905Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-10T03:50:58.708 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.532985298Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-10T03:50:58.708 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.533429029Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-10T03:50:58.708 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.534186545Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-10T03:50:58.709 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.534533453Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-10T03:50:58.709 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.535038414Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-10T03:50:58.709 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.535463562Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-10T03:50:58.709 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.535890559Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-10T03:50:58.709 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.53633416Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-10T03:50:58.709 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.536775886Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-10T03:50:58.709 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.537671575Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-10T03:50:58.709 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.538379167Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-10T03:50:58.709 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.539166501Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-10T03:50:58.709 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.539651284Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-10T03:50:58.709 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.540462878Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-10T03:50:58.709 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.541359856Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-10T03:50:58.709 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.54177536Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-10T03:50:58.710 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.542733494Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-10T03:50:58.710 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.543329284Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-10T03:50:58.710 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.544302035Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-10T03:50:58.710 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.545038489Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-10T03:50:58.710 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.546615223Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-10T03:50:58.710 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.547034819Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-10T03:50:58.710 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.547889392Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-10T03:50:58.710 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.548870736Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-10T03:50:58.710 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.54956032Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-10T03:50:58.710 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.549871155Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-10T03:50:58.710 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.550560448Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-10T03:50:58.711 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.551296045Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-10T03:50:58.711 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.55212557Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-10T03:50:58.711 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.552978034Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-10T03:50:58.711 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.553304598Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-10T03:50:58.711 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.554100507Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-10T03:50:58.711 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.554554297Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-10T03:50:58.711 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.554873534Z level=info msg="Executing migration" id="Move region to single row" 2024-03-10T03:50:58.711 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.555292634Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-10T03:50:58.711 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.557888981Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-10T03:50:58.711 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.558409691Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-10T03:50:58.711 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.558823019Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-10T03:50:58.711 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.559259534Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-10T03:50:58.711 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.561785996Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-10T03:50:58.711 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.564591002Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-10T03:50:58.711 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.567585397Z level=info msg="Executing migration" id="create test_data table" 2024-03-10T03:50:58.712 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.568045375Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-10T03:50:58.712 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.570719325Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-10T03:50:58.712 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.573527268Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-10T03:50:58.712 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.576297414Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-10T03:50:58.712 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.57690075Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-10T03:50:58.712 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.579656779Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-10T03:50:58.712 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.582355152Z level=info msg="Executing migration" id="create team table" 2024-03-10T03:50:58.712 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.585292481Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-10T03:50:58.712 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.585757314Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-10T03:50:58.712 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.588507178Z level=info msg="Executing migration" id="create team member table" 2024-03-10T03:50:58.712 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.59133164Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-10T03:50:58.712 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.594049849Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-10T03:50:58.712 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.59447933Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-10T03:50:58.712 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.597259996Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-10T03:50:58.713 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.600112189Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-10T03:50:58.713 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.602854838Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-10T03:50:58.713 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.603765088Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-10T03:50:58.713 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.606525224Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-10T03:50:58.713 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.609213543Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-10T03:50:58.713 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.611828523Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-10T03:50:58.713 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.612359075Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-10T03:50:58.713 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.61489766Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-10T03:50:58.713 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.61750892Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-10T03:50:58.713 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.62025577Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-10T03:50:58.713 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.62293305Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-10T03:50:58.713 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.625634248Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-10T03:50:58.713 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.626387272Z level=info msg="Executing migration" id="create tag table" 2024-03-10T03:50:58.714 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.6273077Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-10T03:50:58.714 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.628002136Z level=info msg="Executing migration" id="create login attempt table" 2024-03-10T03:50:58.714 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.628917314Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-10T03:50:58.714 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.629636571Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-10T03:50:58.714 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.6304435Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-10T03:50:58.714 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.632313787Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-10T03:50:58.714 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.632725624Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-10T03:50:58.714 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.63351845Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-10T03:50:58.714 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.634415699Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-10T03:50:58.714 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.635122087Z level=info msg="Executing migration" id="create user auth table" 2024-03-10T03:50:58.714 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.636078585Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-10T03:50:58.714 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.636734874Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-10T03:50:58.714 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.637487338Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-10T03:50:58.714 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.638307523Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-10T03:50:58.715 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.639049409Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-10T03:50:58.715 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.639842356Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-10T03:50:58.715 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.640708944Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-10T03:50:58.715 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.641689239Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-10T03:50:58.715 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.642612805Z level=info msg="Executing migration" id="create server_lock table" 2024-03-10T03:50:58.715 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.643549818Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-10T03:50:58.715 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.644402054Z level=info msg="Executing migration" id="create user auth token table" 2024-03-10T03:50:58.715 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.645115641Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-10T03:50:58.715 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.646056529Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-10T03:50:58.715 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.646871547Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-10T03:50:58.715 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.647859141Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-10T03:50:58.715 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.648873002Z level=info msg="Executing migration" id="create cache_data table" 2024-03-10T03:50:58.715 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.649695036Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-10T03:50:58.715 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.650548072Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-10T03:50:58.716 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.651401632Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-10T03:50:58.716 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.65224378Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-10T03:50:58.716 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.653094569Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-10T03:50:58.716 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.654056045Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-10T03:50:58.716 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.654784889Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-10T03:50:58.716 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.655536274Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-10T03:50:58.716 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.658500468Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-10T03:50:58.716 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.661461401Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-10T03:50:58.716 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.66423751Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-10T03:50:58.716 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.667102819Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-10T03:50:58.716 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.67010515Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-10T03:50:58.716 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.672957816Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-10T03:50:58.716 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.675915791Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-10T03:50:58.716 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.678778257Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-10T03:50:58.717 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.681465141Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-10T03:50:58.717 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.684271333Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-10T03:50:58.717 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.687128462Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-10T03:50:58.717 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.689795261Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-10T03:50:58.717 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.692574988Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-10T03:50:58.717 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.695280037Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-10T03:50:58.717 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.698140434Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-10T03:50:58.717 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.700732775Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-10T03:50:58.717 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.703483915Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-10T03:50:58.863 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:50:58.866 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-10T03:50:58.952 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.706169105Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-10T03:50:58.952 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.708799682Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-10T03:50:58.952 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.711471175Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-10T03:50:58.952 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.714908751Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-10T03:50:58.952 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.718199542Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-10T03:50:58.952 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.720970252Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-10T03:50:58.952 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.723744711Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-10T03:50:58.952 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.726579605Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-10T03:50:58.952 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.729240706Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-10T03:50:58.952 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.731701349Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-10T03:50:58.952 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.73444199Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-10T03:50:58.952 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.737104141Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-10T03:50:58.953 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.739844565Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-10T03:50:58.953 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.742647251Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-10T03:50:58.953 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.745078193Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-10T03:50:58.953 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.747846156Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-10T03:50:58.953 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.750404659Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-10T03:50:58.953 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.753207166Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-10T03:50:58.954 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.754028825Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-10T03:50:58.954 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.754852623Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-10T03:50:58.954 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.755629516Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-10T03:50:58.954 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.757982738Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-10T03:50:58.954 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.760555354Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-10T03:50:58.954 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.763496066Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-10T03:50:58.954 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.766435236Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-10T03:50:58.954 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.769314232Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-10T03:50:58.954 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.772055573Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-10T03:50:58.954 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.772355549Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-10T03:50:58.955 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.775202303Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-10T03:50:58.955 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.778196227Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-10T03:50:58.955 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.78092432Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-10T03:50:58.955 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.78386908Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-10T03:50:58.955 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.786500634Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-10T03:50:58.955 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.78936797Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-10T03:50:58.955 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.792249329Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-10T03:50:58.955 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.795045706Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-10T03:50:58.955 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.797645251Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-10T03:50:58.955 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.800338776Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-10T03:50:58.955 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.803094316Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-10T03:50:58.955 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.805788608Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-10T03:50:58.955 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.808445968Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-10T03:50:58.955 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.81104442Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-10T03:50:58.956 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.813761457Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-10T03:50:58.956 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.816609303Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-10T03:50:58.956 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.819045588Z level=info msg="Executing migration" id="create alert_image table" 2024-03-10T03:50:58.956 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.821539324Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-10T03:50:58.956 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.824309294Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-10T03:50:58.956 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.826998445Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-10T03:50:58.956 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.827498187Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-10T03:50:58.956 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.829574599Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-10T03:50:58.956 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.829650981Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-10T03:50:58.956 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.829966135Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-10T03:50:58.956 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.832360228Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-10T03:50:58.956 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.835169316Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-10T03:50:58.957 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.83535239Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-10T03:50:58.957 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.835467997Z level=info msg="alerts found to migrate" alerts=0 2024-03-10T03:50:58.957 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.837807936Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-10T03:50:58.957 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.840363497Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-10T03:50:58.957 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.841113784Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-10T03:50:58.957 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.842129401Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-10T03:50:58.957 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.842967727Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-10T03:50:58.957 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.8436031Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-10T03:50:58.957 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.8444342Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-10T03:50:58.957 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.845052656Z level=info msg="Executing migration" id="create data_keys table" 2024-03-10T03:50:58.957 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.845523772Z level=info msg="Executing migration" id="create secrets table" 2024-03-10T03:50:58.957 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.846425084Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-10T03:50:58.957 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.851005558Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-10T03:50:58.957 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.851959208Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-10T03:50:58.958 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.85283909Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-10T03:50:58.958 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.857093354Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-10T03:50:58.958 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.861348727Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-10T03:50:58.958 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.862090188Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-10T03:50:58.959 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.862570097Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-10T03:50:58.959 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.862897562Z level=info msg="Executing migration" id="create permission table" 2024-03-10T03:50:58.959 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.863322249Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-10T03:50:58.959 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.863764428Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-10T03:50:58.959 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.864229605Z level=info msg="Executing migration" id="create role table" 2024-03-10T03:50:58.959 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.864662215Z level=info msg="Executing migration" id="add column display_name" 2024-03-10T03:50:58.959 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.865608111Z level=info msg="Executing migration" id="add column group_name" 2024-03-10T03:50:58.959 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.866572176Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-10T03:50:58.959 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.867036158Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-10T03:50:58.959 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.867491265Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-10T03:50:58.959 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.867979079Z level=info msg="Executing migration" id="create team role table" 2024-03-10T03:50:58.959 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.868390414Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-10T03:50:58.959 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.868834446Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-10T03:50:58.959 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.869316752Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-10T03:50:58.960 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.869765832Z level=info msg="Executing migration" id="create user role table" 2024-03-10T03:50:58.960 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.870178976Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-10T03:50:58.960 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.870627467Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-10T03:50:58.960 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.871081469Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-10T03:50:58.960 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.871537404Z level=info msg="Executing migration" id="create builtin role table" 2024-03-10T03:50:58.960 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.871971335Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-10T03:50:58.960 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.872768438Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-10T03:50:58.960 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.873250313Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-10T03:50:58.960 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.874237569Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-10T03:50:58.960 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.874724547Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-10T03:50:58.960 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.875529752Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-10T03:50:58.960 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.875989416Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-10T03:50:58.960 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.876435382Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-10T03:50:58.960 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.876831944Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-10T03:50:58.960 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.877312757Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-10T03:50:58.960 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.878294608Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-10T03:50:58.961 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.878705154Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-10T03:50:58.961 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.879160699Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-10T03:50:58.961 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.87947153Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-10T03:50:58.961 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.879883411Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-10T03:50:58.961 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.880365968Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-10T03:50:58.961 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.880739796Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-10T03:50:58.961 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.881064114Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-10T03:50:58.961 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.881443568Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-10T03:50:58.961 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.881838499Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-10T03:50:58.961 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.882293512Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-10T03:50:58.961 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.883283965Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-10T03:50:58.961 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.88360217Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-10T03:50:58.961 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.884020496Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-10T03:50:58.961 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.884467218Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-10T03:50:58.961 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.884913944Z level=info msg="Executing migration" id="add correlation config column" 2024-03-10T03:50:58.961 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.885970972Z level=info msg="Executing migration" id="create entity_events table" 2024-03-10T03:50:58.962 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.886415076Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-10T03:50:58.962 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.886846257Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-10T03:50:58.962 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.88695766Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-10T03:50:58.962 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.88726208Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-10T03:50:58.962 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.887366349Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-10T03:50:58.962 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.887659267Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-10T03:50:58.962 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.888528832Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-10T03:50:58.962 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.889374505Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-10T03:50:58.962 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.89030506Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-10T03:50:58.962 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.891104795Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-10T03:50:58.962 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.891947256Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-10T03:50:58.962 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.892827819Z level=info msg="Executing migration" id="Drop public config table" 2024-03-10T03:50:58.962 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.893825454Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-10T03:50:58.962 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.894530145Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-10T03:50:58.962 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.895437977Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-10T03:50:58.963 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.896099689Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-10T03:50:58.963 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.896528629Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-10T03:50:58.963 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.89950601Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-10T03:50:58.963 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.900529682Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-10T03:50:58.963 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.901577603Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-10T03:50:58.963 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.902639165Z level=info msg="Executing migration" id="add share column" 2024-03-10T03:50:58.963 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.903760932Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-10T03:50:58.963 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.904617685Z level=info msg="Executing migration" id="create file table" 2024-03-10T03:50:58.963 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.905585211Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-10T03:50:58.963 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.908434293Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-10T03:50:58.963 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.911073042Z level=info msg="Executing migration" id="create file_meta table" 2024-03-10T03:50:58.963 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.911519771Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-10T03:50:58.963 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.914682449Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-10T03:50:58.964 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.91714016Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-10T03:50:58.964 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.920043148Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-10T03:50:58.964 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.922919893Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-10T03:50:58.964 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.925754024Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-10T03:50:58.964 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.928571601Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-10T03:50:58.964 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.931292496Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-10T03:50:58.964 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.934076717Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-10T03:50:58.964 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.9369117Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-10T03:50:58.964 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.939873053Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-10T03:50:58.964 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.942663954Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-10T03:50:58.964 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.945291591Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-10T03:50:58.964 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.948113976Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-10T03:50:58.964 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.951027909Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-10T03:50:59.253 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.964622336Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-10T03:50:59.253 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.967479257Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-10T03:50:59.253 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.9681124Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-10T03:50:59.253 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.971468959Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-10T03:50:59.253 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.97233046Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-10T03:50:59.253 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.973115212Z level=info msg="Executing migration" id="create folder table" 2024-03-10T03:50:59.253 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.97397067Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-10T03:50:59.253 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.974838809Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-10T03:50:59.253 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.975481899Z level=info msg="Executing migration" id="Update folder title length" 2024-03-10T03:50:59.253 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.976294886Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-10T03:50:59.253 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=migrator t=2024-03-10T03:50:58.976753235Z level=info msg="migrations completed" performed=485 skipped=0 duration=705.969348ms 2024-03-10T03:50:59.253 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=sqlstore t=2024-03-10T03:50:58.977165255Z level=info msg="Created default organization" 2024-03-10T03:50:59.253 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=plugin.loader t=2024-03-10T03:50:59.031713655Z level=info msg="Plugin registered" pluginID=input 2024-03-10T03:50:59.254 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=plugin.loader t=2024-03-10T03:50:59.058251836Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-10T03:50:59.254 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=plugin.loader t=2024-03-10T03:50:59.058275205Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-10T03:50:59.254 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=secrets t=2024-03-10T03:50:59.058308449Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-10T03:50:59.254 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=query_data t=2024-03-10T03:50:59.060395894Z level=info msg="Query Service initialization" 2024-03-10T03:50:59.254 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=live.push_http t=2024-03-10T03:50:59.063055889Z level=info msg="Live Push Gateway initialization" 2024-03-10T03:50:59.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:59 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:59.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:59 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:59.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:59 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:59.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:59 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:59.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:59 smithi006 ceph-mon[25256]: Deploying daemon node-exporter.a on smithi006 2024-03-10T03:50:59.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:50:59 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:59.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:59 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:59.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:59 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:59.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:59 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:59.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:59 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:59.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:59 smithi006 ceph-mon[31842]: Deploying daemon node-exporter.a on smithi006 2024-03-10T03:50:59.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:50:59 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:59.590 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:59.590 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:59.590 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:59.590 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:59.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-mon[27181]: Deploying daemon node-exporter.a on smithi006 2024-03-10T03:50:59.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:50:59.621 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-03-10T03:50:58.578473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1349,"num_read_kb":1160,"num_write":873,"num_write_kb":4027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":880,"ondisk_log_size":880,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749928448,"kb_used":244912,"kb_used_data":7964,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749683536,"statfs":{"total":767926730752,"available":767675940864,"internally_reserved":0,"allocated":8155136,"data_stored":4374962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003310"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768031+0000","last_change":"2024-03-10T03:49:31.232589+0000","last_active":"2024-03-10T03:50:08.768031+0000","last_peered":"2024-03-10T03:50:08.768031+0000","last_clean":"2024-03-10T03:50:08.768031+0000","last_became_active":"2024-03-10T03:49:31.232085+0000","last_became_peered":"2024-03-10T03:49:31.232085+0000","last_unstale":"2024-03-10T03:50:08.768031+0000","last_undegraded":"2024-03-10T03:50:08.768031+0000","last_fullsized":"2024-03-10T03:50:08.768031+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:55:14.797471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558040+0000","last_change":"2024-03-10T03:49:29.226081+0000","last_active":"2024-03-10T03:50:08.558040+0000","last_peered":"2024-03-10T03:50:08.558040+0000","last_clean":"2024-03-10T03:50:08.558040+0000","last_became_active":"2024-03-10T03:49:29.224282+0000","last_became_peered":"2024-03-10T03:49:29.224282+0000","last_unstale":"2024-03-10T03:50:08.558040+0000","last_undegraded":"2024-03-10T03:50:08.558040+0000","last_fullsized":"2024-03-10T03:50:08.558040+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:41:20.019010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557705+0000","last_change":"2024-03-10T03:49:25.210328+0000","last_active":"2024-03-10T03:50:08.557705+0000","last_peered":"2024-03-10T03:50:08.557705+0000","last_clean":"2024-03-10T03:50:08.557705+0000","last_became_active":"2024-03-10T03:49:25.210044+0000","last_became_peered":"2024-03-10T03:49:25.210044+0000","last_unstale":"2024-03-10T03:50:08.557705+0000","last_undegraded":"2024-03-10T03:50:08.557705+0000","last_fullsized":"2024-03-10T03:50:08.557705+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:10:10.217037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'15","reported_seq":53,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767735+0000","last_change":"2024-03-10T03:49:27.209875+0000","last_active":"2024-03-10T03:50:08.767735+0000","last_peered":"2024-03-10T03:50:08.767735+0000","last_clean":"2024-03-10T03:50:08.767735+0000","last_became_active":"2024-03-10T03:49:27.209614+0000","last_became_peered":"2024-03-10T03:49:27.209614+0000","last_unstale":"2024-03-10T03:50:08.767735+0000","last_undegraded":"2024-03-10T03:50:08.767735+0000","last_fullsized":"2024-03-10T03:50:08.767735+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:34:36.322499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767955+0000","last_change":"2024-03-10T03:49:31.304892+0000","last_active":"2024-03-10T03:50:08.767955+0000","last_peered":"2024-03-10T03:50:08.767955+0000","last_clean":"2024-03-10T03:50:08.767955+0000","last_became_active":"2024-03-10T03:49:31.304507+0000","last_became_peered":"2024-03-10T03:49:31.304507+0000","last_unstale":"2024-03-10T03:50:08.767955+0000","last_undegraded":"2024-03-10T03:50:08.767955+0000","last_fullsized":"2024-03-10T03:50:08.767955+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:05:16.899828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767875+0000","last_change":"2024-03-10T03:49:29.224589+0000","last_active":"2024-03-10T03:50:08.767875+0000","last_peered":"2024-03-10T03:50:08.767875+0000","last_clean":"2024-03-10T03:50:08.767875+0000","last_became_active":"2024-03-10T03:49:29.224294+0000","last_became_peered":"2024-03-10T03:49:29.224294+0000","last_unstale":"2024-03-10T03:50:08.767875+0000","last_undegraded":"2024-03-10T03:50:08.767875+0000","last_fullsized":"2024-03-10T03:50:08.767875+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:40:28.442969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768413+0000","last_change":"2024-03-10T03:49:25.210910+0000","last_active":"2024-03-10T03:50:08.768413+0000","last_peered":"2024-03-10T03:50:08.768413+0000","last_clean":"2024-03-10T03:50:08.768413+0000","last_became_active":"2024-03-10T03:49:25.210608+0000","last_became_peered":"2024-03-10T03:49:25.210608+0000","last_unstale":"2024-03-10T03:50:08.768413+0000","last_undegraded":"2024-03-10T03:50:08.768413+0000","last_fullsized":"2024-03-10T03:50:08.768413+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:18:26.211937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'39","reported_seq":98,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767764+0000","last_change":"2024-03-10T03:49:27.218079+0000","last_active":"2024-03-10T03:50:08.767764+0000","last_peered":"2024-03-10T03:50:08.767764+0000","last_clean":"2024-03-10T03:50:08.767764+0000","last_became_active":"2024-03-10T03:49:27.217768+0000","last_became_peered":"2024-03-10T03:49:27.217768+0000","last_unstale":"2024-03-10T03:50:08.767764+0000","last_undegraded":"2024-03-10T03:50:08.767764+0000","last_fullsized":"2024-03-10T03:50:08.767764+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:22:22.767568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.556456+0000","last_change":"2024-03-10T03:49:31.235460+0000","last_active":"2024-03-10T03:50:08.556456+0000","last_peered":"2024-03-10T03:50:08.556456+0000","last_clean":"2024-03-10T03:50:08.556456+0000","last_became_active":"2024-03-10T03:49:31.234630+0000","last_became_peered":"2024-03-10T03:49:31.234630+0000","last_unstale":"2024-03-10T03:50:08.556456+0000","last_undegraded":"2024-03-10T03:50:08.556456+0000","last_fullsized":"2024-03-10T03:50:08.556456+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:46:41.107159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676610+0000","last_change":"2024-03-10T03:49:29.226991+0000","last_active":"2024-03-10T03:50:08.676610+0000","last_peered":"2024-03-10T03:50:08.676610+0000","last_clean":"2024-03-10T03:50:08.676610+0000","last_became_active":"2024-03-10T03:49:29.226687+0000","last_became_peered":"2024-03-10T03:49:29.226687+0000","last_unstale":"2024-03-10T03:50:08.676610+0000","last_undegraded":"2024-03-10T03:50:08.676610+0000","last_fullsized":"2024-03-10T03:50:08.676610+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:05:18.531132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676681+0000","last_change":"2024-03-10T03:49:25.207650+0000","last_active":"2024-03-10T03:50:08.676681+0000","last_peered":"2024-03-10T03:50:08.676681+0000","last_clean":"2024-03-10T03:50:08.676681+0000","last_became_active":"2024-03-10T03:49:25.207012+0000","last_became_peered":"2024-03-10T03:49:25.207012+0000","last_unstale":"2024-03-10T03:50:08.676681+0000","last_undegraded":"2024-03-10T03:50:08.676681+0000","last_fullsized":"2024-03-10T03:50:08.676681+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:09:03.478081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'23","reported_seq":67,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768262+0000","last_change":"2024-03-10T03:49:27.214213+0000","last_active":"2024-03-10T03:50:08.768262+0000","last_peered":"2024-03-10T03:50:08.768262+0000","last_clean":"2024-03-10T03:50:08.768262+0000","last_became_active":"2024-03-10T03:49:27.213911+0000","last_became_peered":"2024-03-10T03:49:27.213911+0000","last_unstale":"2024-03-10T03:50:08.768262+0000","last_undegraded":"2024-03-10T03:50:08.768262+0000","last_fullsized":"2024-03-10T03:50:08.768262+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:17:26.503750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676354+0000","last_change":"2024-03-10T03:49:31.239545+0000","last_active":"2024-03-10T03:50:08.676354+0000","last_peered":"2024-03-10T03:50:08.676354+0000","last_clean":"2024-03-10T03:50:08.676354+0000","last_became_active":"2024-03-10T03:49:31.239233+0000","last_became_peered":"2024-03-10T03:49:31.239233+0000","last_unstale":"2024-03-10T03:50:08.676354+0000","last_undegraded":"2024-03-10T03:50:08.676354+0000","last_fullsized":"2024-03-10T03:50:08.676354+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:10:12.278049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557914+0000","last_change":"2024-03-10T03:49:29.227820+0000","last_active":"2024-03-10T03:50:08.557914+0000","last_peered":"2024-03-10T03:50:08.557914+0000","last_clean":"2024-03-10T03:50:08.557914+0000","last_became_active":"2024-03-10T03:49:29.227341+0000","last_became_peered":"2024-03-10T03:49:29.227341+0000","last_unstale":"2024-03-10T03:50:08.557914+0000","last_undegraded":"2024-03-10T03:50:08.557914+0000","last_fullsized":"2024-03-10T03:50:08.557914+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:25:02.019917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676638+0000","last_change":"2024-03-10T03:49:25.208025+0000","last_active":"2024-03-10T03:50:08.676638+0000","last_peered":"2024-03-10T03:50:08.676638+0000","last_clean":"2024-03-10T03:50:08.676638+0000","last_became_active":"2024-03-10T03:49:25.207633+0000","last_became_peered":"2024-03-10T03:49:25.207633+0000","last_unstale":"2024-03-10T03:50:08.676638+0000","last_undegraded":"2024-03-10T03:50:08.676638+0000","last_fullsized":"2024-03-10T03:50:08.676638+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T14:29:10.525250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"69'11","reported_seq":47,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768337+0000","last_change":"2024-03-10T03:49:27.214164+0000","last_active":"2024-03-10T03:50:08.768337+0000","last_peered":"2024-03-10T03:50:08.768337+0000","last_clean":"2024-03-10T03:50:08.768337+0000","last_became_active":"2024-03-10T03:49:27.213892+0000","last_became_peered":"2024-03-10T03:49:27.213892+0000","last_unstale":"2024-03-10T03:50:08.768337+0000","last_undegraded":"2024-03-10T03:50:08.768337+0000","last_fullsized":"2024-03-10T03:50:08.768337+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:55:10.609679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768772+0000","last_change":"2024-03-10T03:49:31.303196+0000","last_active":"2024-03-10T03:50:08.768772+0000","last_peered":"2024-03-10T03:50:08.768772+0000","last_clean":"2024-03-10T03:50:08.768772+0000","last_became_active":"2024-03-10T03:49:31.302859+0000","last_became_peered":"2024-03-10T03:49:31.302859+0000","last_unstale":"2024-03-10T03:50:08.768772+0000","last_undegraded":"2024-03-10T03:50:08.768772+0000","last_fullsized":"2024-03-10T03:50:08.768772+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:17:17.940789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557371+0000","last_change":"2024-03-10T03:49:29.225990+0000","last_active":"2024-03-10T03:50:08.557371+0000","last_peered":"2024-03-10T03:50:08.557371+0000","last_clean":"2024-03-10T03:50:08.557371+0000","last_became_active":"2024-03-10T03:49:29.225657+0000","last_became_peered":"2024-03-10T03:49:29.225657+0000","last_unstale":"2024-03-10T03:50:08.557371+0000","last_undegraded":"2024-03-10T03:50:08.557371+0000","last_fullsized":"2024-03-10T03:50:08.557371+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:36:36.627070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557033+0000","last_change":"2024-03-10T03:49:25.207095+0000","last_active":"2024-03-10T03:50:08.557033+0000","last_peered":"2024-03-10T03:50:08.557033+0000","last_clean":"2024-03-10T03:50:08.557033+0000","last_became_active":"2024-03-10T03:49:25.206787+0000","last_became_peered":"2024-03-10T03:49:25.206787+0000","last_unstale":"2024-03-10T03:50:08.557033+0000","last_undegraded":"2024-03-10T03:50:08.557033+0000","last_fullsized":"2024-03-10T03:50:08.557033+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:05:36.111051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.686140+0000","last_change":"2024-03-10T03:49:27.217917+0000","last_active":"2024-03-10T03:50:08.686140+0000","last_peered":"2024-03-10T03:50:08.686140+0000","last_clean":"2024-03-10T03:50:08.686140+0000","last_became_active":"2024-03-10T03:49:27.217616+0000","last_became_peered":"2024-03-10T03:49:27.217616+0000","last_unstale":"2024-03-10T03:50:08.686140+0000","last_undegraded":"2024-03-10T03:50:08.686140+0000","last_fullsized":"2024-03-10T03:50:08.686140+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T13:04:45.259002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768720+0000","last_change":"2024-03-10T03:49:31.303033+0000","last_active":"2024-03-10T03:50:08.768720+0000","last_peered":"2024-03-10T03:50:08.768720+0000","last_clean":"2024-03-10T03:50:08.768720+0000","last_became_active":"2024-03-10T03:49:31.302693+0000","last_became_peered":"2024-03-10T03:49:31.302693+0000","last_unstale":"2024-03-10T03:50:08.768720+0000","last_undegraded":"2024-03-10T03:50:08.768720+0000","last_fullsized":"2024-03-10T03:50:08.768720+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:57:37.472778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768195+0000","last_change":"2024-03-10T03:49:29.220278+0000","last_active":"2024-03-10T03:50:08.768195+0000","last_peered":"2024-03-10T03:50:08.768195+0000","last_clean":"2024-03-10T03:50:08.768195+0000","last_became_active":"2024-03-10T03:49:29.219542+0000","last_became_peered":"2024-03-10T03:49:29.219542+0000","last_unstale":"2024-03-10T03:50:08.768195+0000","last_undegraded":"2024-03-10T03:50:08.768195+0000","last_fullsized":"2024-03-10T03:50:08.768195+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:10:23.476404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768286+0000","last_change":"2024-03-10T03:49:25.209244+0000","last_active":"2024-03-10T03:50:08.768286+0000","last_peered":"2024-03-10T03:50:08.768286+0000","last_clean":"2024-03-10T03:50:08.768286+0000","last_became_active":"2024-03-10T03:49:25.208976+0000","last_became_peered":"2024-03-10T03:49:25.208976+0000","last_unstale":"2024-03-10T03:50:08.768286+0000","last_undegraded":"2024-03-10T03:50:08.768286+0000","last_fullsized":"2024-03-10T03:50:08.768286+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:55:03.017441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'18","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768403+0000","last_change":"2024-03-10T03:49:27.221148+0000","last_active":"2024-03-10T03:50:08.768403+0000","last_peered":"2024-03-10T03:50:08.768403+0000","last_clean":"2024-03-10T03:50:08.768403+0000","last_became_active":"2024-03-10T03:49:27.220793+0000","last_became_peered":"2024-03-10T03:49:27.220793+0000","last_unstale":"2024-03-10T03:50:08.768403+0000","last_undegraded":"2024-03-10T03:50:08.768403+0000","last_fullsized":"2024-03-10T03:50:08.768403+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:08:25.158012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":19,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558523+0000","last_change":"2024-03-10T03:49:31.234581+0000","last_active":"2024-03-10T03:50:08.558523+0000","last_peered":"2024-03-10T03:50:08.558523+0000","last_clean":"2024-03-10T03:50:08.558523+0000","last_became_active":"2024-03-10T03:49:31.234331+0000","last_became_peered":"2024-03-10T03:49:31.234331+0000","last_unstale":"2024-03-10T03:50:08.558523+0000","last_undegraded":"2024-03-10T03:50:08.558523+0000","last_fullsized":"2024-03-10T03:50:08.558523+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T14:14:57.766139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"70'11","reported_seq":53,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:45.271828+0000","last_change":"2024-03-10T03:49:29.223130+0000","last_active":"2024-03-10T03:50:45.271828+0000","last_peered":"2024-03-10T03:50:45.271828+0000","last_clean":"2024-03-10T03:50:45.271828+0000","last_became_active":"2024-03-10T03:49:29.222801+0000","last_became_peered":"2024-03-10T03:49:29.222801+0000","last_unstale":"2024-03-10T03:50:45.271828+0000","last_undegraded":"2024-03-10T03:50:45.271828+0000","last_fullsized":"2024-03-10T03:50:45.271828+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:58:55.411306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":30,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558151+0000","last_change":"2024-03-10T03:49:25.209690+0000","last_active":"2024-03-10T03:50:08.558151+0000","last_peered":"2024-03-10T03:50:08.558151+0000","last_clean":"2024-03-10T03:50:08.558151+0000","last_became_active":"2024-03-10T03:49:25.209411+0000","last_became_peered":"2024-03-10T03:49:25.209411+0000","last_unstale":"2024-03-10T03:50:08.558151+0000","last_undegraded":"2024-03-10T03:50:08.558151+0000","last_fullsized":"2024-03-10T03:50:08.558151+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T15:07:12.056710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"69'16","reported_seq":52,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.556995+0000","last_change":"2024-03-10T03:49:27.226494+0000","last_active":"2024-03-10T03:50:08.556995+0000","last_peered":"2024-03-10T03:50:08.556995+0000","last_clean":"2024-03-10T03:50:08.556995+0000","last_became_active":"2024-03-10T03:49:27.226193+0000","last_became_peered":"2024-03-10T03:49:27.226193+0000","last_unstale":"2024-03-10T03:50:08.556995+0000","last_undegraded":"2024-03-10T03:50:08.556995+0000","last_fullsized":"2024-03-10T03:50:08.556995+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T13:58:19.445874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557533+0000","last_change":"2024-03-10T03:49:31.231756+0000","last_active":"2024-03-10T03:50:08.557533+0000","last_peered":"2024-03-10T03:50:08.557533+0000","last_clean":"2024-03-10T03:50:08.557533+0000","last_became_active":"2024-03-10T03:49:31.231396+0000","last_became_peered":"2024-03-10T03:49:31.231396+0000","last_unstale":"2024-03-10T03:50:08.557533+0000","last_undegraded":"2024-03-10T03:50:08.557533+0000","last_fullsized":"2024-03-10T03:50:08.557533+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:07:37.738114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557324+0000","last_change":"2024-03-10T03:49:29.221814+0000","last_active":"2024-03-10T03:50:08.557324+0000","last_peered":"2024-03-10T03:50:08.557324+0000","last_clean":"2024-03-10T03:50:08.557324+0000","last_became_active":"2024-03-10T03:49:29.221518+0000","last_became_peered":"2024-03-10T03:49:29.221518+0000","last_unstale":"2024-03-10T03:50:08.557324+0000","last_undegraded":"2024-03-10T03:50:08.557324+0000","last_fullsized":"2024-03-10T03:50:08.557324+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:41:24.206929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767256+0000","last_change":"2024-03-10T03:49:25.214747+0000","last_active":"2024-03-10T03:50:08.767256+0000","last_peered":"2024-03-10T03:50:08.767256+0000","last_clean":"2024-03-10T03:50:08.767256+0000","last_became_active":"2024-03-10T03:49:25.214471+0000","last_became_peered":"2024-03-10T03:49:25.214471+0000","last_unstale":"2024-03-10T03:50:08.767256+0000","last_undegraded":"2024-03-10T03:50:08.767256+0000","last_fullsized":"2024-03-10T03:50:08.767256+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T15:21:53.263960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'17","reported_seq":56,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767574+0000","last_change":"2024-03-10T03:49:27.220041+0000","last_active":"2024-03-10T03:50:08.767574+0000","last_peered":"2024-03-10T03:50:08.767574+0000","last_clean":"2024-03-10T03:50:08.767574+0000","last_became_active":"2024-03-10T03:49:27.219709+0000","last_became_peered":"2024-03-10T03:49:27.219709+0000","last_unstale":"2024-03-10T03:50:08.767574+0000","last_undegraded":"2024-03-10T03:50:08.767574+0000","last_fullsized":"2024-03-10T03:50:08.767574+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T15:22:04.462073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557822+0000","last_change":"2024-03-10T03:49:31.237845+0000","last_active":"2024-03-10T03:50:08.557822+0000","last_peered":"2024-03-10T03:50:08.557822+0000","last_clean":"2024-03-10T03:50:08.557822+0000","last_became_active":"2024-03-10T03:49:31.237518+0000","last_became_peered":"2024-03-10T03:49:31.237518+0000","last_unstale":"2024-03-10T03:50:08.557822+0000","last_undegraded":"2024-03-10T03:50:08.557822+0000","last_fullsized":"2024-03-10T03:50:08.557822+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:51:07.465866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767249+0000","last_change":"2024-03-10T03:49:29.229795+0000","last_active":"2024-03-10T03:50:08.767249+0000","last_peered":"2024-03-10T03:50:08.767249+0000","last_clean":"2024-03-10T03:50:08.767249+0000","last_became_active":"2024-03-10T03:49:29.229496+0000","last_became_peered":"2024-03-10T03:49:29.229496+0000","last_unstale":"2024-03-10T03:50:08.767249+0000","last_undegraded":"2024-03-10T03:50:08.767249+0000","last_fullsized":"2024-03-10T03:50:08.767249+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:38:17.789071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557789+0000","last_change":"2024-03-10T03:49:25.216659+0000","last_active":"2024-03-10T03:50:08.557789+0000","last_peered":"2024-03-10T03:50:08.557789+0000","last_clean":"2024-03-10T03:50:08.557789+0000","last_became_active":"2024-03-10T03:49:25.216334+0000","last_became_peered":"2024-03-10T03:49:25.216334+0000","last_unstale":"2024-03-10T03:50:08.557789+0000","last_undegraded":"2024-03-10T03:50:08.557789+0000","last_fullsized":"2024-03-10T03:50:08.557789+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T13:02:37.295580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"69'4","reported_seq":31,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767328+0000","last_change":"2024-03-10T03:49:27.220567+0000","last_active":"2024-03-10T03:50:08.767328+0000","last_peered":"2024-03-10T03:50:08.767328+0000","last_clean":"2024-03-10T03:50:08.767328+0000","last_became_active":"2024-03-10T03:49:27.220255+0000","last_became_peered":"2024-03-10T03:49:27.220255+0000","last_unstale":"2024-03-10T03:50:08.767328+0000","last_undegraded":"2024-03-10T03:50:08.767328+0000","last_fullsized":"2024-03-10T03:50:08.767328+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T13:51:27.826682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767720+0000","last_change":"2024-03-10T03:49:31.304998+0000","last_active":"2024-03-10T03:50:08.767720+0000","last_peered":"2024-03-10T03:50:08.767720+0000","last_clean":"2024-03-10T03:50:08.767720+0000","last_became_active":"2024-03-10T03:49:31.304515+0000","last_became_peered":"2024-03-10T03:49:31.304515+0000","last_unstale":"2024-03-10T03:50:08.767720+0000","last_undegraded":"2024-03-10T03:50:08.767720+0000","last_fullsized":"2024-03-10T03:50:08.767720+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:01:01.378335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558567+0000","last_change":"2024-03-10T03:49:29.228581+0000","last_active":"2024-03-10T03:50:08.558567+0000","last_peered":"2024-03-10T03:50:08.558567+0000","last_clean":"2024-03-10T03:50:08.558567+0000","last_became_active":"2024-03-10T03:49:29.228279+0000","last_became_peered":"2024-03-10T03:49:29.228279+0000","last_unstale":"2024-03-10T03:50:08.558567+0000","last_undegraded":"2024-03-10T03:50:08.558567+0000","last_fullsized":"2024-03-10T03:50:08.558567+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:21:49.987568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676863+0000","last_change":"2024-03-10T03:49:25.204605+0000","last_active":"2024-03-10T03:50:08.676863+0000","last_peered":"2024-03-10T03:50:08.676863+0000","last_clean":"2024-03-10T03:50:08.676863+0000","last_became_active":"2024-03-10T03:49:25.204342+0000","last_became_peered":"2024-03-10T03:49:25.204342+0000","last_unstale":"2024-03-10T03:50:08.676863+0000","last_undegraded":"2024-03-10T03:50:08.676863+0000","last_fullsized":"2024-03-10T03:50:08.676863+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:08:03.853948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"69'17","reported_seq":56,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767552+0000","last_change":"2024-03-10T03:49:27.224564+0000","last_active":"2024-03-10T03:50:08.767552+0000","last_peered":"2024-03-10T03:50:08.767552+0000","last_clean":"2024-03-10T03:50:08.767552+0000","last_became_active":"2024-03-10T03:49:27.224268+0000","last_became_peered":"2024-03-10T03:49:27.224268+0000","last_unstale":"2024-03-10T03:50:08.767552+0000","last_undegraded":"2024-03-10T03:50:08.767552+0000","last_fullsized":"2024-03-10T03:50:08.767552+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:06:09.308204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.555904+0000","last_change":"2024-03-10T03:49:31.232315+0000","last_active":"2024-03-10T03:50:08.555904+0000","last_peered":"2024-03-10T03:50:08.555904+0000","last_clean":"2024-03-10T03:50:08.555904+0000","last_became_active":"2024-03-10T03:49:31.231981+0000","last_became_peered":"2024-03-10T03:49:31.231981+0000","last_unstale":"2024-03-10T03:50:08.555904+0000","last_undegraded":"2024-03-10T03:50:08.555904+0000","last_fullsized":"2024-03-10T03:50:08.555904+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:34:16.673037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767961+0000","last_change":"2024-03-10T03:49:29.222667+0000","last_active":"2024-03-10T03:50:08.767961+0000","last_peered":"2024-03-10T03:50:08.767961+0000","last_clean":"2024-03-10T03:50:08.767961+0000","last_became_active":"2024-03-10T03:49:29.222378+0000","last_became_peered":"2024-03-10T03:49:29.222378+0000","last_unstale":"2024-03-10T03:50:08.767961+0000","last_undegraded":"2024-03-10T03:50:08.767961+0000","last_fullsized":"2024-03-10T03:50:08.767961+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:12:44.452463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558651+0000","last_change":"2024-03-10T03:49:25.214331+0000","last_active":"2024-03-10T03:50:08.558651+0000","last_peered":"2024-03-10T03:50:08.558651+0000","last_clean":"2024-03-10T03:50:08.558651+0000","last_became_active":"2024-03-10T03:49:25.214039+0000","last_became_peered":"2024-03-10T03:49:25.214039+0000","last_unstale":"2024-03-10T03:50:08.558651+0000","last_undegraded":"2024-03-10T03:50:08.558651+0000","last_fullsized":"2024-03-10T03:50:08.558651+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T13:20:23.361735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"69'9","reported_seq":41,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.559840+0000","last_change":"2024-03-10T03:49:27.213754+0000","last_active":"2024-03-10T03:50:08.559840+0000","last_peered":"2024-03-10T03:50:08.559840+0000","last_clean":"2024-03-10T03:50:08.559840+0000","last_became_active":"2024-03-10T03:49:27.213363+0000","last_became_peered":"2024-03-10T03:49:27.213363+0000","last_unstale":"2024-03-10T03:50:08.559840+0000","last_undegraded":"2024-03-10T03:50:08.559840+0000","last_fullsized":"2024-03-10T03:50:08.559840+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:05:49.339755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767363+0000","last_change":"2024-03-10T03:49:31.241899+0000","last_active":"2024-03-10T03:50:08.767363+0000","last_peered":"2024-03-10T03:50:08.767363+0000","last_clean":"2024-03-10T03:50:08.767363+0000","last_became_active":"2024-03-10T03:49:31.241459+0000","last_became_peered":"2024-03-10T03:49:31.241459+0000","last_unstale":"2024-03-10T03:50:08.767363+0000","last_undegraded":"2024-03-10T03:50:08.767363+0000","last_fullsized":"2024-03-10T03:50:08.767363+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:52:23.376147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.559599+0000","last_change":"2024-03-10T03:49:29.224799+0000","last_active":"2024-03-10T03:50:08.559599+0000","last_peered":"2024-03-10T03:50:08.559599+0000","last_clean":"2024-03-10T03:50:08.559599+0000","last_became_active":"2024-03-10T03:49:29.224448+0000","last_became_peered":"2024-03-10T03:49:29.224448+0000","last_unstale":"2024-03-10T03:50:08.559599+0000","last_undegraded":"2024-03-10T03:50:08.559599+0000","last_fullsized":"2024-03-10T03:50:08.559599+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T13:35:21.264435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768457+0000","last_change":"2024-03-10T03:49:25.211384+0000","last_active":"2024-03-10T03:50:08.768457+0000","last_peered":"2024-03-10T03:50:08.768457+0000","last_clean":"2024-03-10T03:50:08.768457+0000","last_became_active":"2024-03-10T03:49:25.211105+0000","last_became_peered":"2024-03-10T03:49:25.211105+0000","last_unstale":"2024-03-10T03:50:08.768457+0000","last_undegraded":"2024-03-10T03:50:08.768457+0000","last_fullsized":"2024-03-10T03:50:08.768457+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:41:09.133320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"69'11","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768518+0000","last_change":"2024-03-10T03:49:27.208982+0000","last_active":"2024-03-10T03:50:08.768518+0000","last_peered":"2024-03-10T03:50:08.768518+0000","last_clean":"2024-03-10T03:50:08.768518+0000","last_became_active":"2024-03-10T03:49:27.208682+0000","last_became_peered":"2024-03-10T03:49:27.208682+0000","last_unstale":"2024-03-10T03:50:08.768518+0000","last_undegraded":"2024-03-10T03:50:08.768518+0000","last_fullsized":"2024-03-10T03:50:08.768518+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:15:07.281839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.556670+0000","last_change":"2024-03-10T03:49:31.242771+0000","last_active":"2024-03-10T03:50:08.556670+0000","last_peered":"2024-03-10T03:50:08.556670+0000","last_clean":"2024-03-10T03:50:08.556670+0000","last_became_active":"2024-03-10T03:49:31.242375+0000","last_became_peered":"2024-03-10T03:49:31.242375+0000","last_unstale":"2024-03-10T03:50:08.556670+0000","last_undegraded":"2024-03-10T03:50:08.556670+0000","last_fullsized":"2024-03-10T03:50:08.556670+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T14:51:17.780789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"70'11","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:45.271813+0000","last_change":"2024-03-10T03:49:29.225588+0000","last_active":"2024-03-10T03:50:45.271813+0000","last_peered":"2024-03-10T03:50:45.271813+0000","last_clean":"2024-03-10T03:50:45.271813+0000","last_became_active":"2024-03-10T03:49:29.225281+0000","last_became_peered":"2024-03-10T03:49:29.225281+0000","last_unstale":"2024-03-10T03:50:45.271813+0000","last_undegraded":"2024-03-10T03:50:45.271813+0000","last_fullsized":"2024-03-10T03:50:45.271813+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T13:00:58.748745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557993+0000","last_change":"2024-03-10T03:49:25.210071+0000","last_active":"2024-03-10T03:50:08.557993+0000","last_peered":"2024-03-10T03:50:08.557993+0000","last_clean":"2024-03-10T03:50:08.557993+0000","last_became_active":"2024-03-10T03:49:25.209806+0000","last_became_peered":"2024-03-10T03:49:25.209806+0000","last_unstale":"2024-03-10T03:50:08.557993+0000","last_undegraded":"2024-03-10T03:50:08.557993+0000","last_fullsized":"2024-03-10T03:50:08.557993+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:05:18.614686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"69'22","reported_seq":64,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768001+0000","last_change":"2024-03-10T03:49:27.227462+0000","last_active":"2024-03-10T03:50:08.768001+0000","last_peered":"2024-03-10T03:50:08.768001+0000","last_clean":"2024-03-10T03:50:08.768001+0000","last_became_active":"2024-03-10T03:49:27.227085+0000","last_became_peered":"2024-03-10T03:49:27.227085+0000","last_unstale":"2024-03-10T03:50:08.768001+0000","last_undegraded":"2024-03-10T03:50:08.768001+0000","last_fullsized":"2024-03-10T03:50:08.768001+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:42:03.647651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768686+0000","last_change":"2024-03-10T03:49:31.237861+0000","last_active":"2024-03-10T03:50:08.768686+0000","last_peered":"2024-03-10T03:50:08.768686+0000","last_clean":"2024-03-10T03:50:08.768686+0000","last_became_active":"2024-03-10T03:49:31.237584+0000","last_became_peered":"2024-03-10T03:49:31.237584+0000","last_unstale":"2024-03-10T03:50:08.768686+0000","last_undegraded":"2024-03-10T03:50:08.768686+0000","last_fullsized":"2024-03-10T03:50:08.768686+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:12:19.353438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"70'11","reported_seq":53,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:45.271883+0000","last_change":"2024-03-10T03:49:29.221165+0000","last_active":"2024-03-10T03:50:45.271883+0000","last_peered":"2024-03-10T03:50:45.271883+0000","last_clean":"2024-03-10T03:50:45.271883+0000","last_became_active":"2024-03-10T03:49:29.220865+0000","last_became_peered":"2024-03-10T03:49:29.220865+0000","last_unstale":"2024-03-10T03:50:45.271883+0000","last_undegraded":"2024-03-10T03:50:45.271883+0000","last_fullsized":"2024-03-10T03:50:45.271883+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T15:28:08.462333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.556275+0000","last_change":"2024-03-10T03:49:25.216824+0000","last_active":"2024-03-10T03:50:08.556275+0000","last_peered":"2024-03-10T03:50:08.556275+0000","last_clean":"2024-03-10T03:50:08.556275+0000","last_became_active":"2024-03-10T03:49:25.216569+0000","last_became_peered":"2024-03-10T03:49:25.216569+0000","last_unstale":"2024-03-10T03:50:08.556275+0000","last_undegraded":"2024-03-10T03:50:08.556275+0000","last_fullsized":"2024-03-10T03:50:08.556275+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:23:48.509540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"69'15","reported_seq":53,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676967+0000","last_change":"2024-03-10T03:49:27.225582+0000","last_active":"2024-03-10T03:50:08.676967+0000","last_peered":"2024-03-10T03:50:08.676967+0000","last_clean":"2024-03-10T03:50:08.676967+0000","last_became_active":"2024-03-10T03:49:27.225278+0000","last_became_peered":"2024-03-10T03:49:27.225278+0000","last_unstale":"2024-03-10T03:50:08.676967+0000","last_undegraded":"2024-03-10T03:50:08.676967+0000","last_fullsized":"2024-03-10T03:50:08.676967+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:57:10.878877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.686483+0000","last_change":"2024-03-10T03:49:31.234849+0000","last_active":"2024-03-10T03:50:08.686483+0000","last_peered":"2024-03-10T03:50:08.686483+0000","last_clean":"2024-03-10T03:50:08.686483+0000","last_became_active":"2024-03-10T03:49:31.234438+0000","last_became_peered":"2024-03-10T03:49:31.234438+0000","last_unstale":"2024-03-10T03:50:08.686483+0000","last_undegraded":"2024-03-10T03:50:08.686483+0000","last_fullsized":"2024-03-10T03:50:08.686483+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T14:49:05.413593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768691+0000","last_change":"2024-03-10T03:49:29.227165+0000","last_active":"2024-03-10T03:50:08.768691+0000","last_peered":"2024-03-10T03:50:08.768691+0000","last_clean":"2024-03-10T03:50:08.768691+0000","last_became_active":"2024-03-10T03:49:29.226840+0000","last_became_peered":"2024-03-10T03:49:29.226840+0000","last_unstale":"2024-03-10T03:50:08.768691+0000","last_undegraded":"2024-03-10T03:50:08.768691+0000","last_fullsized":"2024-03-10T03:50:08.768691+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T13:05:47.034971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"62'1","reported_seq":37,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558012+0000","last_change":"2024-03-10T03:49:25.212208+0000","last_active":"2024-03-10T03:50:08.558012+0000","last_peered":"2024-03-10T03:50:08.558012+0000","last_clean":"2024-03-10T03:50:08.558012+0000","last_became_active":"2024-03-10T03:49:25.211908+0000","last_became_peered":"2024-03-10T03:49:25.211908+0000","last_unstale":"2024-03-10T03:50:08.558012+0000","last_undegraded":"2024-03-10T03:50:08.558012+0000","last_fullsized":"2024-03-10T03:50:08.558012+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:46:41.351712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'21","reported_seq":65,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557137+0000","last_change":"2024-03-10T03:49:27.225782+0000","last_active":"2024-03-10T03:50:08.557137+0000","last_peered":"2024-03-10T03:50:08.557137+0000","last_clean":"2024-03-10T03:50:08.557137+0000","last_became_active":"2024-03-10T03:49:27.225487+0000","last_became_peered":"2024-03-10T03:49:27.225487+0000","last_unstale":"2024-03-10T03:50:08.557137+0000","last_undegraded":"2024-03-10T03:50:08.557137+0000","last_fullsized":"2024-03-10T03:50:08.557137+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T13:23:13.259430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558548+0000","last_change":"2024-03-10T03:49:31.305179+0000","last_active":"2024-03-10T03:50:08.558548+0000","last_peered":"2024-03-10T03:50:08.558548+0000","last_clean":"2024-03-10T03:50:08.558548+0000","last_became_active":"2024-03-10T03:49:31.304779+0000","last_became_peered":"2024-03-10T03:49:31.304779+0000","last_unstale":"2024-03-10T03:50:08.558548+0000","last_undegraded":"2024-03-10T03:50:08.558548+0000","last_fullsized":"2024-03-10T03:50:08.558548+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:16:31.779939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676909+0000","last_change":"2024-03-10T03:49:29.222395+0000","last_active":"2024-03-10T03:50:08.676909+0000","last_peered":"2024-03-10T03:50:08.676909+0000","last_clean":"2024-03-10T03:50:08.676909+0000","last_became_active":"2024-03-10T03:49:29.222089+0000","last_became_peered":"2024-03-10T03:49:29.222089+0000","last_unstale":"2024-03-10T03:50:08.676909+0000","last_undegraded":"2024-03-10T03:50:08.676909+0000","last_fullsized":"2024-03-10T03:50:08.676909+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T15:26:51.165830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767194+0000","last_change":"2024-03-10T03:49:25.214883+0000","last_active":"2024-03-10T03:50:08.767194+0000","last_peered":"2024-03-10T03:50:08.767194+0000","last_clean":"2024-03-10T03:50:08.767194+0000","last_became_active":"2024-03-10T03:49:25.214593+0000","last_became_peered":"2024-03-10T03:49:25.214593+0000","last_unstale":"2024-03-10T03:50:08.767194+0000","last_undegraded":"2024-03-10T03:50:08.767194+0000","last_fullsized":"2024-03-10T03:50:08.767194+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:27:40.787890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"69'12","reported_seq":46,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768783+0000","last_change":"2024-03-10T03:49:27.226530+0000","last_active":"2024-03-10T03:50:08.768783+0000","last_peered":"2024-03-10T03:50:08.768783+0000","last_clean":"2024-03-10T03:50:08.768783+0000","last_became_active":"2024-03-10T03:49:27.226227+0000","last_became_peered":"2024-03-10T03:49:27.226227+0000","last_unstale":"2024-03-10T03:50:08.768783+0000","last_undegraded":"2024-03-10T03:50:08.768783+0000","last_fullsized":"2024-03-10T03:50:08.768783+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:52:29.571582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676818+0000","last_change":"2024-03-10T03:49:31.302968+0000","last_active":"2024-03-10T03:50:08.676818+0000","last_peered":"2024-03-10T03:50:08.676818+0000","last_clean":"2024-03-10T03:50:08.676818+0000","last_became_active":"2024-03-10T03:49:31.302495+0000","last_became_peered":"2024-03-10T03:49:31.302495+0000","last_unstale":"2024-03-10T03:50:08.676818+0000","last_undegraded":"2024-03-10T03:50:08.676818+0000","last_fullsized":"2024-03-10T03:50:08.676818+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:59:51.712808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.559132+0000","last_change":"2024-03-10T03:49:25.213952+0000","last_active":"2024-03-10T03:50:08.559132+0000","last_peered":"2024-03-10T03:50:08.559132+0000","last_clean":"2024-03-10T03:50:08.559132+0000","last_became_active":"2024-03-10T03:49:25.213652+0000","last_became_peered":"2024-03-10T03:49:25.213652+0000","last_unstale":"2024-03-10T03:50:08.559132+0000","last_undegraded":"2024-03-10T03:50:08.559132+0000","last_fullsized":"2024-03-10T03:50:08.559132+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T14:20:41.107876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"69'27","reported_seq":74,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676379+0000","last_change":"2024-03-10T03:49:27.226178+0000","last_active":"2024-03-10T03:50:08.676379+0000","last_peered":"2024-03-10T03:50:08.676379+0000","last_clean":"2024-03-10T03:50:08.676379+0000","last_became_active":"2024-03-10T03:49:27.225871+0000","last_became_peered":"2024-03-10T03:49:27.225871+0000","last_unstale":"2024-03-10T03:50:08.676379+0000","last_undegraded":"2024-03-10T03:50:08.676379+0000","last_fullsized":"2024-03-10T03:50:08.676379+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T15:01:50.869781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"70'11","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:45.271854+0000","last_change":"2024-03-10T03:49:29.230442+0000","last_active":"2024-03-10T03:50:45.271854+0000","last_peered":"2024-03-10T03:50:45.271854+0000","last_clean":"2024-03-10T03:50:45.271854+0000","last_became_active":"2024-03-10T03:49:29.230156+0000","last_became_peered":"2024-03-10T03:49:29.230156+0000","last_unstale":"2024-03-10T03:50:45.271854+0000","last_undegraded":"2024-03-10T03:50:45.271854+0000","last_fullsized":"2024-03-10T03:50:45.271854+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:36:09.029657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767865+0000","last_change":"2024-03-10T03:49:31.238755+0000","last_active":"2024-03-10T03:50:08.767865+0000","last_peered":"2024-03-10T03:50:08.767865+0000","last_clean":"2024-03-10T03:50:08.767865+0000","last_became_active":"2024-03-10T03:49:31.237221+0000","last_became_peered":"2024-03-10T03:49:31.237221+0000","last_unstale":"2024-03-10T03:50:08.767865+0000","last_undegraded":"2024-03-10T03:50:08.767865+0000","last_fullsized":"2024-03-10T03:50:08.767865+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:49:11.201929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558286+0000","last_change":"2024-03-10T03:49:25.210399+0000","last_active":"2024-03-10T03:50:08.558286+0000","last_peered":"2024-03-10T03:50:08.558286+0000","last_clean":"2024-03-10T03:50:08.558286+0000","last_became_active":"2024-03-10T03:49:25.209796+0000","last_became_peered":"2024-03-10T03:49:25.209796+0000","last_unstale":"2024-03-10T03:50:08.558286+0000","last_undegraded":"2024-03-10T03:50:08.558286+0000","last_fullsized":"2024-03-10T03:50:08.558286+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:14:18.333004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"69'24","reported_seq":72,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768094+0000","last_change":"2024-03-10T03:49:27.217870+0000","last_active":"2024-03-10T03:50:08.768094+0000","last_peered":"2024-03-10T03:50:08.768094+0000","last_clean":"2024-03-10T03:50:08.768094+0000","last_became_active":"2024-03-10T03:49:27.217694+0000","last_became_peered":"2024-03-10T03:49:27.217694+0000","last_unstale":"2024-03-10T03:50:08.768094+0000","last_undegraded":"2024-03-10T03:50:08.768094+0000","last_fullsized":"2024-03-10T03:50:08.768094+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:05:26.866353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.685681+0000","last_change":"2024-03-10T03:49:29.221281+0000","last_active":"2024-03-10T03:50:08.685681+0000","last_peered":"2024-03-10T03:50:08.685681+0000","last_clean":"2024-03-10T03:50:08.685681+0000","last_became_active":"2024-03-10T03:49:29.220944+0000","last_became_peered":"2024-03-10T03:49:29.220944+0000","last_unstale":"2024-03-10T03:50:08.685681+0000","last_undegraded":"2024-03-10T03:50:08.685681+0000","last_fullsized":"2024-03-10T03:50:08.685681+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:58:44.147234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557618+0000","last_change":"2024-03-10T03:49:31.240182+0000","last_active":"2024-03-10T03:50:08.557618+0000","last_peered":"2024-03-10T03:50:08.557618+0000","last_clean":"2024-03-10T03:50:08.557618+0000","last_became_active":"2024-03-10T03:49:31.239844+0000","last_became_peered":"2024-03-10T03:49:31.239844+0000","last_unstale":"2024-03-10T03:50:08.557618+0000","last_undegraded":"2024-03-10T03:50:08.557618+0000","last_fullsized":"2024-03-10T03:50:08.557618+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:03:38.448627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.686034+0000","last_change":"2024-03-10T03:49:29.226447+0000","last_active":"2024-03-10T03:50:08.686034+0000","last_peered":"2024-03-10T03:50:08.686034+0000","last_clean":"2024-03-10T03:50:08.686034+0000","last_became_active":"2024-03-10T03:49:29.226165+0000","last_became_peered":"2024-03-10T03:49:29.226165+0000","last_unstale":"2024-03-10T03:50:08.686034+0000","last_undegraded":"2024-03-10T03:50:08.686034+0000","last_fullsized":"2024-03-10T03:50:08.686034+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:08:06.601491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558861+0000","last_change":"2024-03-10T03:49:25.208976+0000","last_active":"2024-03-10T03:50:08.558861+0000","last_peered":"2024-03-10T03:50:08.558861+0000","last_clean":"2024-03-10T03:50:08.558861+0000","last_became_active":"2024-03-10T03:49:25.208386+0000","last_became_peered":"2024-03-10T03:49:25.208386+0000","last_unstale":"2024-03-10T03:50:08.558861+0000","last_undegraded":"2024-03-10T03:50:08.558861+0000","last_fullsized":"2024-03-10T03:50:08.558861+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:20:13.465647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"69'25","reported_seq":68,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767577+0000","last_change":"2024-03-10T03:49:27.225629+0000","last_active":"2024-03-10T03:50:08.767577+0000","last_peered":"2024-03-10T03:50:08.767577+0000","last_clean":"2024-03-10T03:50:08.767577+0000","last_became_active":"2024-03-10T03:49:27.225286+0000","last_became_peered":"2024-03-10T03:49:27.225286+0000","last_unstale":"2024-03-10T03:50:08.767577+0000","last_undegraded":"2024-03-10T03:50:08.767577+0000","last_fullsized":"2024-03-10T03:50:08.767577+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:04:53.452163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.556067+0000","last_change":"2024-03-10T03:49:31.233965+0000","last_active":"2024-03-10T03:50:08.556067+0000","last_peered":"2024-03-10T03:50:08.556067+0000","last_clean":"2024-03-10T03:50:08.556067+0000","last_became_active":"2024-03-10T03:49:31.233033+0000","last_became_peered":"2024-03-10T03:49:31.233033+0000","last_unstale":"2024-03-10T03:50:08.556067+0000","last_undegraded":"2024-03-10T03:50:08.556067+0000","last_fullsized":"2024-03-10T03:50:08.556067+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T13:31:56.437954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.685999+0000","last_change":"2024-03-10T03:49:29.218933+0000","last_active":"2024-03-10T03:50:08.685999+0000","last_peered":"2024-03-10T03:50:08.685999+0000","last_clean":"2024-03-10T03:50:08.685999+0000","last_became_active":"2024-03-10T03:49:29.218638+0000","last_became_peered":"2024-03-10T03:49:29.218638+0000","last_unstale":"2024-03-10T03:50:08.685999+0000","last_undegraded":"2024-03-10T03:50:08.685999+0000","last_fullsized":"2024-03-10T03:50:08.685999+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:34:56.549956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676668+0000","last_change":"2024-03-10T03:49:25.211190+0000","last_active":"2024-03-10T03:50:08.676668+0000","last_peered":"2024-03-10T03:50:08.676668+0000","last_clean":"2024-03-10T03:50:08.676668+0000","last_became_active":"2024-03-10T03:49:25.210855+0000","last_became_peered":"2024-03-10T03:49:25.210855+0000","last_unstale":"2024-03-10T03:50:08.676668+0000","last_undegraded":"2024-03-10T03:50:08.676668+0000","last_fullsized":"2024-03-10T03:50:08.676668+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:30:07.502998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"69'15","reported_seq":53,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557441+0000","last_change":"2024-03-10T03:49:27.214715+0000","last_active":"2024-03-10T03:50:08.557441+0000","last_peered":"2024-03-10T03:50:08.557441+0000","last_clean":"2024-03-10T03:50:08.557441+0000","last_became_active":"2024-03-10T03:49:27.214438+0000","last_became_peered":"2024-03-10T03:49:27.214438+0000","last_unstale":"2024-03-10T03:50:08.557441+0000","last_undegraded":"2024-03-10T03:50:08.557441+0000","last_fullsized":"2024-03-10T03:50:08.557441+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T14:19:49.905441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768156+0000","last_change":"2024-03-10T03:49:31.238061+0000","last_active":"2024-03-10T03:50:08.768156+0000","last_peered":"2024-03-10T03:50:08.768156+0000","last_clean":"2024-03-10T03:50:08.768156+0000","last_became_active":"2024-03-10T03:49:31.237686+0000","last_became_peered":"2024-03-10T03:49:31.237686+0000","last_unstale":"2024-03-10T03:50:08.768156+0000","last_undegraded":"2024-03-10T03:50:08.768156+0000","last_fullsized":"2024-03-10T03:50:08.768156+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:54:12.652235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"66'2","reported_seq":38,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676524+0000","last_change":"2024-03-10T03:49:29.223417+0000","last_active":"2024-03-10T03:50:08.676524+0000","last_peered":"2024-03-10T03:50:08.676524+0000","last_clean":"2024-03-10T03:50:08.676524+0000","last_became_active":"2024-03-10T03:49:25.204053+0000","last_became_peered":"2024-03-10T03:49:25.204053+0000","last_unstale":"2024-03-10T03:50:08.676524+0000","last_undegraded":"2024-03-10T03:50:08.676524+0000","last_fullsized":"2024-03-10T03:50:08.676524+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:44:55.447963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049149550000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768055+0000","last_change":"2024-03-10T03:49:25.214213+0000","last_active":"2024-03-10T03:50:08.768055+0000","last_peered":"2024-03-10T03:50:08.768055+0000","last_clean":"2024-03-10T03:50:08.768055+0000","last_became_active":"2024-03-10T03:49:25.213894+0000","last_became_peered":"2024-03-10T03:49:25.213894+0000","last_unstale":"2024-03-10T03:50:08.768055+0000","last_undegraded":"2024-03-10T03:50:08.768055+0000","last_fullsized":"2024-03-10T03:50:08.768055+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:12:39.847132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"70'36","reported_seq":106,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:45.271820+0000","last_change":"2024-03-10T03:49:27.216890+0000","last_active":"2024-03-10T03:50:45.271820+0000","last_peered":"2024-03-10T03:50:45.271820+0000","last_clean":"2024-03-10T03:50:45.271820+0000","last_became_active":"2024-03-10T03:49:27.216607+0000","last_became_peered":"2024-03-10T03:49:27.216607+0000","last_unstale":"2024-03-10T03:50:45.271820+0000","last_undegraded":"2024-03-10T03:50:45.271820+0000","last_fullsized":"2024-03-10T03:50:45.271820+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:40:46.914229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.556565+0000","last_change":"2024-03-10T03:49:29.223573+0000","last_active":"2024-03-10T03:50:08.556565+0000","last_peered":"2024-03-10T03:50:08.556565+0000","last_clean":"2024-03-10T03:50:08.556565+0000","last_became_active":"2024-03-10T03:49:29.223300+0000","last_became_peered":"2024-03-10T03:49:29.223300+0000","last_unstale":"2024-03-10T03:50:08.556565+0000","last_undegraded":"2024-03-10T03:50:08.556565+0000","last_fullsized":"2024-03-10T03:50:08.556565+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:13:35.609350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.559015+0000","last_change":"2024-03-10T03:49:31.303370+0000","last_active":"2024-03-10T03:50:08.559015+0000","last_peered":"2024-03-10T03:50:08.559015+0000","last_clean":"2024-03-10T03:50:08.559015+0000","last_became_active":"2024-03-10T03:49:31.302937+0000","last_became_peered":"2024-03-10T03:49:31.302937+0000","last_unstale":"2024-03-10T03:50:08.559015+0000","last_undegraded":"2024-03-10T03:50:08.559015+0000","last_fullsized":"2024-03-10T03:50:08.559015+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:48:59.980889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557548+0000","last_change":"2024-03-10T03:49:25.208243+0000","last_active":"2024-03-10T03:50:08.557548+0000","last_peered":"2024-03-10T03:50:08.557548+0000","last_clean":"2024-03-10T03:50:08.557548+0000","last_became_active":"2024-03-10T03:49:25.207949+0000","last_became_peered":"2024-03-10T03:49:25.207949+0000","last_unstale":"2024-03-10T03:50:08.557548+0000","last_undegraded":"2024-03-10T03:50:08.557548+0000","last_fullsized":"2024-03-10T03:50:08.557548+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T14:03:52.412562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":36,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.685882+0000","last_change":"2024-03-10T03:49:29.214944+0000","last_active":"2024-03-10T03:50:08.685882+0000","last_peered":"2024-03-10T03:50:08.685882+0000","last_clean":"2024-03-10T03:50:08.685882+0000","last_became_active":"2024-03-10T03:49:25.212958+0000","last_became_peered":"2024-03-10T03:49:25.212958+0000","last_unstale":"2024-03-10T03:50:08.685882+0000","last_undegraded":"2024-03-10T03:50:08.685882+0000","last_fullsized":"2024-03-10T03:50:08.685882+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:42:22.054366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003673029,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"69'19","reported_seq":64,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558514+0000","last_change":"2024-03-10T03:49:27.217528+0000","last_active":"2024-03-10T03:50:08.558514+0000","last_peered":"2024-03-10T03:50:08.558514+0000","last_clean":"2024-03-10T03:50:08.558514+0000","last_became_active":"2024-03-10T03:49:27.217249+0000","last_became_peered":"2024-03-10T03:49:27.217249+0000","last_unstale":"2024-03-10T03:50:08.558514+0000","last_undegraded":"2024-03-10T03:50:08.558514+0000","last_fullsized":"2024-03-10T03:50:08.558514+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:48:56.758854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.686273+0000","last_change":"2024-03-10T03:49:29.225740+0000","last_active":"2024-03-10T03:50:08.686273+0000","last_peered":"2024-03-10T03:50:08.686273+0000","last_clean":"2024-03-10T03:50:08.686273+0000","last_became_active":"2024-03-10T03:49:29.225430+0000","last_became_peered":"2024-03-10T03:49:29.225430+0000","last_unstale":"2024-03-10T03:50:08.686273+0000","last_undegraded":"2024-03-10T03:50:08.686273+0000","last_fullsized":"2024-03-10T03:50:08.686273+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:18:44.898443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"69'1","reported_seq":18,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676976+0000","last_change":"2024-03-10T03:49:31.239529+0000","last_active":"2024-03-10T03:50:08.676976+0000","last_peered":"2024-03-10T03:50:08.676976+0000","last_clean":"2024-03-10T03:50:08.676976+0000","last_became_active":"2024-03-10T03:49:31.239196+0000","last_became_peered":"2024-03-10T03:49:31.239196+0000","last_unstale":"2024-03-10T03:50:08.676976+0000","last_undegraded":"2024-03-10T03:50:08.676976+0000","last_fullsized":"2024-03-10T03:50:08.676976+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:21:35.680393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"71'188","reported_seq":344,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:10.734018+0000","last_change":"2024-03-10T03:49:12.961905+0000","last_active":"2024-03-10T03:50:10.734018+0000","last_peered":"2024-03-10T03:50:10.734018+0000","last_clean":"2024-03-10T03:50:10.734018+0000","last_became_active":"2024-03-10T03:49:12.961515+0000","last_became_peered":"2024-03-10T03:49:12.961515+0000","last_unstale":"2024-03-10T03:50:10.734018+0000","last_undegraded":"2024-03-10T03:50:10.734018+0000","last_fullsized":"2024-03-10T03:50:10.734018+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:47:01.843240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:47:01.843240+0000","last_clean_scrub_stamp":"2024-03-10T03:47:01.843240+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:25:33.117670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767730+0000","last_change":"2024-03-10T03:49:25.210934+0000","last_active":"2024-03-10T03:50:08.767730+0000","last_peered":"2024-03-10T03:50:08.767730+0000","last_clean":"2024-03-10T03:50:08.767730+0000","last_became_active":"2024-03-10T03:49:25.208887+0000","last_became_peered":"2024-03-10T03:49:25.208887+0000","last_unstale":"2024-03-10T03:50:08.767730+0000","last_undegraded":"2024-03-10T03:50:08.767730+0000","last_fullsized":"2024-03-10T03:50:08.767730+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:22:37.687467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"69'16","reported_seq":65,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:45.271813+0000","last_change":"2024-03-10T03:49:27.224799+0000","last_active":"2024-03-10T03:50:45.271813+0000","last_peered":"2024-03-10T03:50:45.271813+0000","last_clean":"2024-03-10T03:50:45.271813+0000","last_became_active":"2024-03-10T03:49:27.224486+0000","last_became_peered":"2024-03-10T03:49:27.224486+0000","last_unstale":"2024-03-10T03:50:45.271813+0000","last_undegraded":"2024-03-10T03:50:45.271813+0000","last_fullsized":"2024-03-10T03:50:45.271813+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:12:02.696623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558708+0000","last_change":"2024-03-10T03:49:29.231568+0000","last_active":"2024-03-10T03:50:08.558708+0000","last_peered":"2024-03-10T03:50:08.558708+0000","last_clean":"2024-03-10T03:50:08.558708+0000","last_became_active":"2024-03-10T03:49:29.231267+0000","last_became_peered":"2024-03-10T03:49:29.231267+0000","last_unstale":"2024-03-10T03:50:08.558708+0000","last_undegraded":"2024-03-10T03:50:08.558708+0000","last_fullsized":"2024-03-10T03:50:08.558708+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:26:21.829789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.559745+0000","last_change":"2024-03-10T03:49:31.240018+0000","last_active":"2024-03-10T03:50:08.559745+0000","last_peered":"2024-03-10T03:50:08.559745+0000","last_clean":"2024-03-10T03:50:08.559745+0000","last_became_active":"2024-03-10T03:49:31.239699+0000","last_became_peered":"2024-03-10T03:49:31.239699+0000","last_unstale":"2024-03-10T03:50:08.559745+0000","last_undegraded":"2024-03-10T03:50:08.559745+0000","last_fullsized":"2024-03-10T03:50:08.559745+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:58:01.177798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"62'1","reported_seq":37,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.556303+0000","last_change":"2024-03-10T03:49:25.217129+0000","last_active":"2024-03-10T03:50:08.556303+0000","last_peered":"2024-03-10T03:50:08.556303+0000","last_clean":"2024-03-10T03:50:08.556303+0000","last_became_active":"2024-03-10T03:49:25.216788+0000","last_became_peered":"2024-03-10T03:49:25.216788+0000","last_unstale":"2024-03-10T03:50:08.556303+0000","last_undegraded":"2024-03-10T03:50:08.556303+0000","last_fullsized":"2024-03-10T03:50:08.556303+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:08:45.529042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":120,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:52.908361+0000","last_change":"2024-03-10T03:49:29.223404+0000","last_active":"2024-03-10T03:50:52.908361+0000","last_peered":"2024-03-10T03:50:52.908361+0000","last_clean":"2024-03-10T03:50:52.908361+0000","last_became_active":"2024-03-10T03:49:25.210697+0000","last_became_peered":"2024-03-10T03:49:25.210697+0000","last_unstale":"2024-03-10T03:50:52.908361+0000","last_undegraded":"2024-03-10T03:50:52.908361+0000","last_fullsized":"2024-03-10T03:50:52.908361+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:03:44.881734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011382409999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":81,"num_read_kb":76,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"69'12","reported_seq":43,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.556766+0000","last_change":"2024-03-10T03:49:27.214011+0000","last_active":"2024-03-10T03:50:08.556766+0000","last_peered":"2024-03-10T03:50:08.556766+0000","last_clean":"2024-03-10T03:50:08.556766+0000","last_became_active":"2024-03-10T03:49:27.213719+0000","last_became_peered":"2024-03-10T03:49:27.213719+0000","last_unstale":"2024-03-10T03:50:08.556766+0000","last_undegraded":"2024-03-10T03:50:08.556766+0000","last_fullsized":"2024-03-10T03:50:08.556766+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:13:56.642032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.677131+0000","last_change":"2024-03-10T03:49:29.228840+0000","last_active":"2024-03-10T03:50:08.677131+0000","last_peered":"2024-03-10T03:50:08.677131+0000","last_clean":"2024-03-10T03:50:08.677131+0000","last_became_active":"2024-03-10T03:49:29.228546+0000","last_became_peered":"2024-03-10T03:49:29.228546+0000","last_unstale":"2024-03-10T03:50:08.677131+0000","last_undegraded":"2024-03-10T03:50:08.677131+0000","last_fullsized":"2024-03-10T03:50:08.677131+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:11:22.484853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557920+0000","last_change":"2024-03-10T03:49:31.303021+0000","last_active":"2024-03-10T03:50:08.557920+0000","last_peered":"2024-03-10T03:50:08.557920+0000","last_clean":"2024-03-10T03:50:08.557920+0000","last_became_active":"2024-03-10T03:49:31.302599+0000","last_became_peered":"2024-03-10T03:49:31.302599+0000","last_unstale":"2024-03-10T03:50:08.557920+0000","last_undegraded":"2024-03-10T03:50:08.557920+0000","last_fullsized":"2024-03-10T03:50:08.557920+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:35:25.934917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.559996+0000","last_change":"2024-03-10T03:49:25.201942+0000","last_active":"2024-03-10T03:50:08.559996+0000","last_peered":"2024-03-10T03:50:08.559996+0000","last_clean":"2024-03-10T03:50:08.559996+0000","last_became_active":"2024-03-10T03:49:25.201605+0000","last_became_peered":"2024-03-10T03:49:25.201605+0000","last_unstale":"2024-03-10T03:50:08.559996+0000","last_undegraded":"2024-03-10T03:50:08.559996+0000","last_fullsized":"2024-03-10T03:50:08.559996+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:06:06.152147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"69'25","reported_seq":73,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557220+0000","last_change":"2024-03-10T03:49:27.225667+0000","last_active":"2024-03-10T03:50:08.557220+0000","last_peered":"2024-03-10T03:50:08.557220+0000","last_clean":"2024-03-10T03:50:08.557220+0000","last_became_active":"2024-03-10T03:49:27.225224+0000","last_became_peered":"2024-03-10T03:49:27.225224+0000","last_unstale":"2024-03-10T03:50:08.557220+0000","last_undegraded":"2024-03-10T03:50:08.557220+0000","last_fullsized":"2024-03-10T03:50:08.557220+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:03:00.490166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767642+0000","last_change":"2024-03-10T03:49:29.225780+0000","last_active":"2024-03-10T03:50:08.767642+0000","last_peered":"2024-03-10T03:50:08.767642+0000","last_clean":"2024-03-10T03:50:08.767642+0000","last_became_active":"2024-03-10T03:49:29.225488+0000","last_became_peered":"2024-03-10T03:49:29.225488+0000","last_unstale":"2024-03-10T03:50:08.767642+0000","last_undegraded":"2024-03-10T03:50:08.767642+0000","last_fullsized":"2024-03-10T03:50:08.767642+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:54:58.916740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.555911+0000","last_change":"2024-03-10T03:49:31.243419+0000","last_active":"2024-03-10T03:50:08.555911+0000","last_peered":"2024-03-10T03:50:08.555911+0000","last_clean":"2024-03-10T03:50:08.555911+0000","last_became_active":"2024-03-10T03:49:31.243070+0000","last_became_peered":"2024-03-10T03:49:31.243070+0000","last_unstale":"2024-03-10T03:50:08.555911+0000","last_undegraded":"2024-03-10T03:50:08.555911+0000","last_fullsized":"2024-03-10T03:50:08.555911+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:03:13.208872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676802+0000","last_change":"2024-03-10T03:49:25.211684+0000","last_active":"2024-03-10T03:50:08.676802+0000","last_peered":"2024-03-10T03:50:08.676802+0000","last_clean":"2024-03-10T03:50:08.676802+0000","last_became_active":"2024-03-10T03:49:25.210379+0000","last_became_peered":"2024-03-10T03:49:25.210379+0000","last_unstale":"2024-03-10T03:50:08.676802+0000","last_undegraded":"2024-03-10T03:50:08.676802+0000","last_fullsized":"2024-03-10T03:50:08.676802+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:46:09.417306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"69'16","reported_seq":52,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.559705+0000","last_change":"2024-03-10T03:49:27.211176+0000","last_active":"2024-03-10T03:50:08.559705+0000","last_peered":"2024-03-10T03:50:08.559705+0000","last_clean":"2024-03-10T03:50:08.559705+0000","last_became_active":"2024-03-10T03:49:27.210864+0000","last_became_peered":"2024-03-10T03:49:27.210864+0000","last_unstale":"2024-03-10T03:50:08.559705+0000","last_undegraded":"2024-03-10T03:50:08.559705+0000","last_fullsized":"2024-03-10T03:50:08.559705+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:35:32.001856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"70'11","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:45.271833+0000","last_change":"2024-03-10T03:49:29.226373+0000","last_active":"2024-03-10T03:50:45.271833+0000","last_peered":"2024-03-10T03:50:45.271833+0000","last_clean":"2024-03-10T03:50:45.271833+0000","last_became_active":"2024-03-10T03:49:29.226090+0000","last_became_peered":"2024-03-10T03:49:29.226090+0000","last_unstale":"2024-03-10T03:50:45.271833+0000","last_undegraded":"2024-03-10T03:50:45.271833+0000","last_fullsized":"2024-03-10T03:50:45.271833+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:59:07.060353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558344+0000","last_change":"2024-03-10T03:49:31.304476+0000","last_active":"2024-03-10T03:50:08.558344+0000","last_peered":"2024-03-10T03:50:08.558344+0000","last_clean":"2024-03-10T03:50:08.558344+0000","last_became_active":"2024-03-10T03:49:31.304170+0000","last_became_peered":"2024-03-10T03:49:31.304170+0000","last_unstale":"2024-03-10T03:50:08.558344+0000","last_undegraded":"2024-03-10T03:50:08.558344+0000","last_fullsized":"2024-03-10T03:50:08.558344+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:22:35.639526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":30,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.556893+0000","last_change":"2024-03-10T03:49:25.214344+0000","last_active":"2024-03-10T03:50:08.556893+0000","last_peered":"2024-03-10T03:50:08.556893+0000","last_clean":"2024-03-10T03:50:08.556893+0000","last_became_active":"2024-03-10T03:49:25.214034+0000","last_became_peered":"2024-03-10T03:49:25.214034+0000","last_unstale":"2024-03-10T03:50:08.556893+0000","last_undegraded":"2024-03-10T03:50:08.556893+0000","last_fullsized":"2024-03-10T03:50:08.556893+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T15:18:20.722043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"69'20","reported_seq":58,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767972+0000","last_change":"2024-03-10T03:49:27.211589+0000","last_active":"2024-03-10T03:50:08.767972+0000","last_peered":"2024-03-10T03:50:08.767972+0000","last_clean":"2024-03-10T03:50:08.767972+0000","last_became_active":"2024-03-10T03:49:27.211059+0000","last_became_peered":"2024-03-10T03:49:27.211059+0000","last_unstale":"2024-03-10T03:50:08.767972+0000","last_undegraded":"2024-03-10T03:50:08.767972+0000","last_fullsized":"2024-03-10T03:50:08.767972+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:37:14.305807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"70'11","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:45.271868+0000","last_change":"2024-03-10T03:49:29.221225+0000","last_active":"2024-03-10T03:50:45.271868+0000","last_peered":"2024-03-10T03:50:45.271868+0000","last_clean":"2024-03-10T03:50:45.271868+0000","last_became_active":"2024-03-10T03:49:29.220806+0000","last_became_peered":"2024-03-10T03:49:29.220806+0000","last_unstale":"2024-03-10T03:50:45.271868+0000","last_undegraded":"2024-03-10T03:50:45.271868+0000","last_fullsized":"2024-03-10T03:50:45.271868+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:36:51.097675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768123+0000","last_change":"2024-03-10T03:49:31.238598+0000","last_active":"2024-03-10T03:50:08.768123+0000","last_peered":"2024-03-10T03:50:08.768123+0000","last_clean":"2024-03-10T03:50:08.768123+0000","last_became_active":"2024-03-10T03:49:31.238307+0000","last_became_peered":"2024-03-10T03:49:31.238307+0000","last_unstale":"2024-03-10T03:50:08.768123+0000","last_undegraded":"2024-03-10T03:50:08.768123+0000","last_fullsized":"2024-03-10T03:50:08.768123+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:31:59.005442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768599+0000","last_change":"2024-03-10T03:49:25.217775+0000","last_active":"2024-03-10T03:50:08.768599+0000","last_peered":"2024-03-10T03:50:08.768599+0000","last_clean":"2024-03-10T03:50:08.768599+0000","last_became_active":"2024-03-10T03:49:25.217424+0000","last_became_peered":"2024-03-10T03:49:25.217424+0000","last_unstale":"2024-03-10T03:50:08.768599+0000","last_undegraded":"2024-03-10T03:50:08.768599+0000","last_fullsized":"2024-03-10T03:50:08.768599+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:28:15.980479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"69'30","reported_seq":81,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767456+0000","last_change":"2024-03-10T03:49:27.225474+0000","last_active":"2024-03-10T03:50:08.767456+0000","last_peered":"2024-03-10T03:50:08.767456+0000","last_clean":"2024-03-10T03:50:08.767456+0000","last_became_active":"2024-03-10T03:49:27.225207+0000","last_became_peered":"2024-03-10T03:49:27.225207+0000","last_unstale":"2024-03-10T03:50:08.767456+0000","last_undegraded":"2024-03-10T03:50:08.767456+0000","last_fullsized":"2024-03-10T03:50:08.767456+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:35:15.031216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767783+0000","last_change":"2024-03-10T03:49:29.223283+0000","last_active":"2024-03-10T03:50:08.767783+0000","last_peered":"2024-03-10T03:50:08.767783+0000","last_clean":"2024-03-10T03:50:08.767783+0000","last_became_active":"2024-03-10T03:49:29.222977+0000","last_became_peered":"2024-03-10T03:49:29.222977+0000","last_unstale":"2024-03-10T03:50:08.767783+0000","last_undegraded":"2024-03-10T03:50:08.767783+0000","last_fullsized":"2024-03-10T03:50:08.767783+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T15:08:12.040216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676676+0000","last_change":"2024-03-10T03:49:31.231766+0000","last_active":"2024-03-10T03:50:08.676676+0000","last_peered":"2024-03-10T03:50:08.676676+0000","last_clean":"2024-03-10T03:50:08.676676+0000","last_became_active":"2024-03-10T03:49:31.231486+0000","last_became_peered":"2024-03-10T03:49:31.231486+0000","last_unstale":"2024-03-10T03:50:08.676676+0000","last_undegraded":"2024-03-10T03:50:08.676676+0000","last_fullsized":"2024-03-10T03:50:08.676676+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T14:09:55.084176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"70'11","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:45.272072+0000","last_change":"2024-03-10T03:49:29.224677+0000","last_active":"2024-03-10T03:50:45.272072+0000","last_peered":"2024-03-10T03:50:45.272072+0000","last_clean":"2024-03-10T03:50:45.272072+0000","last_became_active":"2024-03-10T03:49:29.224368+0000","last_became_peered":"2024-03-10T03:49:29.224368+0000","last_unstale":"2024-03-10T03:50:45.272072+0000","last_undegraded":"2024-03-10T03:50:45.272072+0000","last_fullsized":"2024-03-10T03:50:45.272072+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:32:13.390081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767376+0000","last_change":"2024-03-10T03:49:25.213828+0000","last_active":"2024-03-10T03:50:08.767376+0000","last_peered":"2024-03-10T03:50:08.767376+0000","last_clean":"2024-03-10T03:50:08.767376+0000","last_became_active":"2024-03-10T03:49:25.213523+0000","last_became_peered":"2024-03-10T03:49:25.213523+0000","last_unstale":"2024-03-10T03:50:08.767376+0000","last_undegraded":"2024-03-10T03:50:08.767376+0000","last_fullsized":"2024-03-10T03:50:08.767376+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:04:01.802567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"69'15","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558700+0000","last_change":"2024-03-10T03:49:27.215761+0000","last_active":"2024-03-10T03:50:08.558700+0000","last_peered":"2024-03-10T03:50:08.558700+0000","last_clean":"2024-03-10T03:50:08.558700+0000","last_became_active":"2024-03-10T03:49:27.215390+0000","last_became_peered":"2024-03-10T03:49:27.215390+0000","last_unstale":"2024-03-10T03:50:08.558700+0000","last_undegraded":"2024-03-10T03:50:08.558700+0000","last_fullsized":"2024-03-10T03:50:08.558700+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:36:27.724770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767910+0000","last_change":"2024-03-10T03:49:31.303033+0000","last_active":"2024-03-10T03:50:08.767910+0000","last_peered":"2024-03-10T03:50:08.767910+0000","last_clean":"2024-03-10T03:50:08.767910+0000","last_became_active":"2024-03-10T03:49:31.302665+0000","last_became_peered":"2024-03-10T03:49:31.302665+0000","last_unstale":"2024-03-10T03:50:08.767910+0000","last_undegraded":"2024-03-10T03:50:08.767910+0000","last_fullsized":"2024-03-10T03:50:08.767910+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:52:53.372407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.677125+0000","last_change":"2024-03-10T03:49:29.231051+0000","last_active":"2024-03-10T03:50:08.677125+0000","last_peered":"2024-03-10T03:50:08.677125+0000","last_clean":"2024-03-10T03:50:08.677125+0000","last_became_active":"2024-03-10T03:49:29.230777+0000","last_became_peered":"2024-03-10T03:49:29.230777+0000","last_unstale":"2024-03-10T03:50:08.677125+0000","last_undegraded":"2024-03-10T03:50:08.677125+0000","last_fullsized":"2024-03-10T03:50:08.677125+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:19:30.050107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767846+0000","last_change":"2024-03-10T03:49:25.210285+0000","last_active":"2024-03-10T03:50:08.767846+0000","last_peered":"2024-03-10T03:50:08.767846+0000","last_clean":"2024-03-10T03:50:08.767846+0000","last_became_active":"2024-03-10T03:49:25.209986+0000","last_became_peered":"2024-03-10T03:49:25.209986+0000","last_unstale":"2024-03-10T03:50:08.767846+0000","last_undegraded":"2024-03-10T03:50:08.767846+0000","last_fullsized":"2024-03-10T03:50:08.767846+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:05:13.072307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"69'17","reported_seq":56,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768750+0000","last_change":"2024-03-10T03:49:27.209375+0000","last_active":"2024-03-10T03:50:08.768750+0000","last_peered":"2024-03-10T03:50:08.768750+0000","last_clean":"2024-03-10T03:50:08.768750+0000","last_became_active":"2024-03-10T03:49:27.209075+0000","last_became_peered":"2024-03-10T03:49:27.209075+0000","last_unstale":"2024-03-10T03:50:08.768750+0000","last_undegraded":"2024-03-10T03:50:08.768750+0000","last_fullsized":"2024-03-10T03:50:08.768750+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:17:50.344799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.686250+0000","last_change":"2024-03-10T03:49:31.240710+0000","last_active":"2024-03-10T03:50:08.686250+0000","last_peered":"2024-03-10T03:50:08.686250+0000","last_clean":"2024-03-10T03:50:08.686250+0000","last_became_active":"2024-03-10T03:49:31.240378+0000","last_became_peered":"2024-03-10T03:49:31.240378+0000","last_unstale":"2024-03-10T03:50:08.686250+0000","last_undegraded":"2024-03-10T03:50:08.686250+0000","last_fullsized":"2024-03-10T03:50:08.686250+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:52:39.392944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.560163+0000","last_change":"2024-03-10T03:49:29.225948+0000","last_active":"2024-03-10T03:50:08.560163+0000","last_peered":"2024-03-10T03:50:08.560163+0000","last_clean":"2024-03-10T03:50:08.560163+0000","last_became_active":"2024-03-10T03:49:29.225621+0000","last_became_peered":"2024-03-10T03:49:29.225621+0000","last_unstale":"2024-03-10T03:50:08.560163+0000","last_undegraded":"2024-03-10T03:50:08.560163+0000","last_fullsized":"2024-03-10T03:50:08.560163+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:54:34.072463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767743+0000","last_change":"2024-03-10T03:49:25.207108+0000","last_active":"2024-03-10T03:50:08.767743+0000","last_peered":"2024-03-10T03:50:08.767743+0000","last_clean":"2024-03-10T03:50:08.767743+0000","last_became_active":"2024-03-10T03:49:25.206813+0000","last_became_peered":"2024-03-10T03:49:25.206813+0000","last_unstale":"2024-03-10T03:50:08.767743+0000","last_undegraded":"2024-03-10T03:50:08.767743+0000","last_fullsized":"2024-03-10T03:50:08.767743+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:18:59.642192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"69'17","reported_seq":53,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768686+0000","last_change":"2024-03-10T03:49:27.207609+0000","last_active":"2024-03-10T03:50:08.768686+0000","last_peered":"2024-03-10T03:50:08.768686+0000","last_clean":"2024-03-10T03:50:08.768686+0000","last_became_active":"2024-03-10T03:49:27.207265+0000","last_became_peered":"2024-03-10T03:49:27.207265+0000","last_unstale":"2024-03-10T03:50:08.768686+0000","last_undegraded":"2024-03-10T03:50:08.768686+0000","last_fullsized":"2024-03-10T03:50:08.768686+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:19:45.912429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768892+0000","last_change":"2024-03-10T03:49:31.238754+0000","last_active":"2024-03-10T03:50:08.768892+0000","last_peered":"2024-03-10T03:50:08.768892+0000","last_clean":"2024-03-10T03:50:08.768892+0000","last_became_active":"2024-03-10T03:49:31.238441+0000","last_became_peered":"2024-03-10T03:49:31.238441+0000","last_unstale":"2024-03-10T03:50:08.768892+0000","last_undegraded":"2024-03-10T03:50:08.768892+0000","last_fullsized":"2024-03-10T03:50:08.768892+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T13:54:31.344185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"70'11","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:45.271846+0000","last_change":"2024-03-10T03:49:29.229294+0000","last_active":"2024-03-10T03:50:45.271846+0000","last_peered":"2024-03-10T03:50:45.271846+0000","last_clean":"2024-03-10T03:50:45.271846+0000","last_became_active":"2024-03-10T03:49:29.228967+0000","last_became_peered":"2024-03-10T03:49:29.228967+0000","last_unstale":"2024-03-10T03:50:45.271846+0000","last_undegraded":"2024-03-10T03:50:45.271846+0000","last_fullsized":"2024-03-10T03:50:45.271846+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:49:59.707748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768053+0000","last_change":"2024-03-10T03:49:25.218892+0000","last_active":"2024-03-10T03:50:08.768053+0000","last_peered":"2024-03-10T03:50:08.768053+0000","last_clean":"2024-03-10T03:50:08.768053+0000","last_became_active":"2024-03-10T03:49:25.218608+0000","last_became_peered":"2024-03-10T03:49:25.218608+0000","last_unstale":"2024-03-10T03:50:08.768053+0000","last_undegraded":"2024-03-10T03:50:08.768053+0000","last_fullsized":"2024-03-10T03:50:08.768053+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T15:04:14.178696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"69'10","reported_seq":40,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768290+0000","last_change":"2024-03-10T03:49:27.217125+0000","last_active":"2024-03-10T03:50:08.768290+0000","last_peered":"2024-03-10T03:50:08.768290+0000","last_clean":"2024-03-10T03:50:08.768290+0000","last_became_active":"2024-03-10T03:49:27.216833+0000","last_became_peered":"2024-03-10T03:49:27.216833+0000","last_unstale":"2024-03-10T03:50:08.768290+0000","last_undegraded":"2024-03-10T03:50:08.768290+0000","last_fullsized":"2024-03-10T03:50:08.768290+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:35:16.280725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":989,"num_read_kb":714,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":589,"ondisk_log_size":589,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":188,"ondisk_log_size":188,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":76,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168600,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":28372,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93712684,"statfs":{"total":95990841344,"available":95961788416,"internally_reserved":0,"allocated":1388544,"data_stored":916599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430236,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32512,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708544,"statfs":{"total":95990841344,"available":95957549056,"internally_reserved":0,"allocated":1368064,"data_stored":915547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31996,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709060,"statfs":{"total":95990841344,"available":95958077440,"internally_reserved":0,"allocated":835584,"data_stored":326089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738429,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":28376,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93712680,"statfs":{"total":95990841344,"available":95961784320,"internally_reserved":0,"allocated":1392640,"data_stored":915864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31928,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709128,"statfs":{"total":95990841344,"available":95958147072,"internally_reserved":0,"allocated":765952,"data_stored":324419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149740,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713240,"statfs":{"total":95990841344,"available":95962357760,"internally_reserved":0,"allocated":815104,"data_stored":325606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986215,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31972,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709084,"statfs":{"total":95990841344,"available":95958102016,"internally_reserved":0,"allocated":811008,"data_stored":326195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724578,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31940,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709116,"statfs":{"total":95990841344,"available":95958134784,"internally_reserved":0,"allocated":778240,"data_stored":324643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-10T03:50:59.628 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph pg dump --format=json 2024-03-10T03:50:59.923 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a[47087]: debug there is no tcmu-runner data available 2024-03-10T03:50:59.923 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=infra.usagestats.collector t=2024-03-10T03:50:59.670533853Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-10T03:50:59.924 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=provisioning.datasources t=2024-03-10T03:50:59.671500519Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-10T03:50:59.924 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=provisioning.datasources t=2024-03-10T03:50:59.682195356Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-10T03:50:59.924 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=provisioning.alerting t=2024-03-10T03:50:59.693094983Z level=info msg="starting to provision alerting" 2024-03-10T03:50:59.924 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=provisioning.alerting t=2024-03-10T03:50:59.693115352Z level=info msg="finished to provision alerting" 2024-03-10T03:50:59.924 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=grafanaStorageLogger t=2024-03-10T03:50:59.693291532Z level=info msg="storage starting" 2024-03-10T03:50:59.924 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=ngalert.state.manager t=2024-03-10T03:50:59.693321852Z level=info msg="Warming state cache for startup" 2024-03-10T03:50:59.924 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=ngalert.state.manager t=2024-03-10T03:50:59.693684492Z level=info msg="State cache has been initialized" states=0 duration=361.602µs 2024-03-10T03:50:59.924 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=ticker t=2024-03-10T03:50:59.693790358Z level=info msg=starting first_tick=2024-03-10T03:51:00Z 2024-03-10T03:50:59.924 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=ngalert.multiorg.alertmanager t=2024-03-10T03:50:59.693819601Z level=info msg="starting MultiOrg Alertmanager" 2024-03-10T03:50:59.924 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=http.server t=2024-03-10T03:50:59.695063369Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-10T03:50:59.924 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=sqlstore.transactions t=2024-03-10T03:50:59.708906916Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-10T03:50:59.924 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=sqlstore.transactions t=2024-03-10T03:50:59.713426667Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-10T03:50:59.924 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=sqlstore.transactions t=2024-03-10T03:50:59.719233936Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-10T03:50:59.924 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=sqlstore.transactions t=2024-03-10T03:50:59.729642369Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-10T03:50:59.925 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=sqlstore.transactions t=2024-03-10T03:50:59.740500844Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-03-10T03:50:59.925 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:50:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a[51276]: logger=sqlstore.transactions t=2024-03-10T03:50:59.752294313Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-10T03:51:00.393 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:51:00.465 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:00 smithi006 ceph-mon[31842]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:51:00.465 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:00 smithi006 ceph-mon[31842]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-10T03:51:00.465 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:00 smithi006 ceph-mon[25256]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:51:00.465 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:00 smithi006 ceph-mon[25256]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-10T03:51:00.590 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:00 smithi071 ceph-mon[27181]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:51:00.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:00 smithi071 ceph-mon[27181]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-10T03:51:01.535 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:01 smithi071 ceph-mon[27181]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:51:01.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:01 smithi006 ceph-mon[25256]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:51:01.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:01 smithi006 ceph-mon[31842]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:51:02.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:02 smithi071 ceph-mon[27181]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:51:02.568 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:51:02.570 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:02 smithi006 ceph-mon[25256]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:51:02.570 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:02 smithi006 ceph-mon[31842]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:51:02.570 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-10T03:51:03.086 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:02 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: ::ffff:172.21.15.71 - - [10/Mar/2024:03:51:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-10T03:51:04.087 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:03 smithi006 bash[61556]: Writing manifest to image destination 2024-03-10T03:51:04.504 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:04 smithi006 ceph-mon[31842]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-10T03:51:04.504 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:04 smithi006 ceph-mon[31842]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:51:04.505 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:04 smithi006 podman[61556]: 2024-03-10 03:51:04.180845971 +0000 UTC m=+3.493012409 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-10T03:51:04.505 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:04 smithi006 ceph-mon[25256]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-10T03:51:04.505 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:04 smithi006 ceph-mon[25256]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:51:04.575 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:04 smithi071 ceph-mon[27181]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-10T03:51:04.575 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:04 smithi071 ceph-mon[27181]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:51:04.801 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-03-10T03:51:00.578796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1354,"num_read_kb":1165,"num_write":873,"num_write_kb":4027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":880,"ondisk_log_size":880,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749928448,"kb_used":244912,"kb_used_data":7964,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749683536,"statfs":{"total":767926730752,"available":767675940864,"internally_reserved":0,"allocated":8155136,"data_stored":4374962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003034"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768031+0000","last_change":"2024-03-10T03:49:31.232589+0000","last_active":"2024-03-10T03:50:08.768031+0000","last_peered":"2024-03-10T03:50:08.768031+0000","last_clean":"2024-03-10T03:50:08.768031+0000","last_became_active":"2024-03-10T03:49:31.232085+0000","last_became_peered":"2024-03-10T03:49:31.232085+0000","last_unstale":"2024-03-10T03:50:08.768031+0000","last_undegraded":"2024-03-10T03:50:08.768031+0000","last_fullsized":"2024-03-10T03:50:08.768031+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:55:14.797471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558040+0000","last_change":"2024-03-10T03:49:29.226081+0000","last_active":"2024-03-10T03:50:08.558040+0000","last_peered":"2024-03-10T03:50:08.558040+0000","last_clean":"2024-03-10T03:50:08.558040+0000","last_became_active":"2024-03-10T03:49:29.224282+0000","last_became_peered":"2024-03-10T03:49:29.224282+0000","last_unstale":"2024-03-10T03:50:08.558040+0000","last_undegraded":"2024-03-10T03:50:08.558040+0000","last_fullsized":"2024-03-10T03:50:08.558040+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:41:20.019010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557705+0000","last_change":"2024-03-10T03:49:25.210328+0000","last_active":"2024-03-10T03:50:08.557705+0000","last_peered":"2024-03-10T03:50:08.557705+0000","last_clean":"2024-03-10T03:50:08.557705+0000","last_became_active":"2024-03-10T03:49:25.210044+0000","last_became_peered":"2024-03-10T03:49:25.210044+0000","last_unstale":"2024-03-10T03:50:08.557705+0000","last_undegraded":"2024-03-10T03:50:08.557705+0000","last_fullsized":"2024-03-10T03:50:08.557705+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:10:10.217037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"69'15","reported_seq":53,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767735+0000","last_change":"2024-03-10T03:49:27.209875+0000","last_active":"2024-03-10T03:50:08.767735+0000","last_peered":"2024-03-10T03:50:08.767735+0000","last_clean":"2024-03-10T03:50:08.767735+0000","last_became_active":"2024-03-10T03:49:27.209614+0000","last_became_peered":"2024-03-10T03:49:27.209614+0000","last_unstale":"2024-03-10T03:50:08.767735+0000","last_undegraded":"2024-03-10T03:50:08.767735+0000","last_fullsized":"2024-03-10T03:50:08.767735+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:34:36.322499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767955+0000","last_change":"2024-03-10T03:49:31.304892+0000","last_active":"2024-03-10T03:50:08.767955+0000","last_peered":"2024-03-10T03:50:08.767955+0000","last_clean":"2024-03-10T03:50:08.767955+0000","last_became_active":"2024-03-10T03:49:31.304507+0000","last_became_peered":"2024-03-10T03:49:31.304507+0000","last_unstale":"2024-03-10T03:50:08.767955+0000","last_undegraded":"2024-03-10T03:50:08.767955+0000","last_fullsized":"2024-03-10T03:50:08.767955+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:05:16.899828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767875+0000","last_change":"2024-03-10T03:49:29.224589+0000","last_active":"2024-03-10T03:50:08.767875+0000","last_peered":"2024-03-10T03:50:08.767875+0000","last_clean":"2024-03-10T03:50:08.767875+0000","last_became_active":"2024-03-10T03:49:29.224294+0000","last_became_peered":"2024-03-10T03:49:29.224294+0000","last_unstale":"2024-03-10T03:50:08.767875+0000","last_undegraded":"2024-03-10T03:50:08.767875+0000","last_fullsized":"2024-03-10T03:50:08.767875+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:40:28.442969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768413+0000","last_change":"2024-03-10T03:49:25.210910+0000","last_active":"2024-03-10T03:50:08.768413+0000","last_peered":"2024-03-10T03:50:08.768413+0000","last_clean":"2024-03-10T03:50:08.768413+0000","last_became_active":"2024-03-10T03:49:25.210608+0000","last_became_peered":"2024-03-10T03:49:25.210608+0000","last_unstale":"2024-03-10T03:50:08.768413+0000","last_undegraded":"2024-03-10T03:50:08.768413+0000","last_fullsized":"2024-03-10T03:50:08.768413+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:18:26.211937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"69'39","reported_seq":98,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767764+0000","last_change":"2024-03-10T03:49:27.218079+0000","last_active":"2024-03-10T03:50:08.767764+0000","last_peered":"2024-03-10T03:50:08.767764+0000","last_clean":"2024-03-10T03:50:08.767764+0000","last_became_active":"2024-03-10T03:49:27.217768+0000","last_became_peered":"2024-03-10T03:49:27.217768+0000","last_unstale":"2024-03-10T03:50:08.767764+0000","last_undegraded":"2024-03-10T03:50:08.767764+0000","last_fullsized":"2024-03-10T03:50:08.767764+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:22:22.767568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.556456+0000","last_change":"2024-03-10T03:49:31.235460+0000","last_active":"2024-03-10T03:50:08.556456+0000","last_peered":"2024-03-10T03:50:08.556456+0000","last_clean":"2024-03-10T03:50:08.556456+0000","last_became_active":"2024-03-10T03:49:31.234630+0000","last_became_peered":"2024-03-10T03:49:31.234630+0000","last_unstale":"2024-03-10T03:50:08.556456+0000","last_undegraded":"2024-03-10T03:50:08.556456+0000","last_fullsized":"2024-03-10T03:50:08.556456+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:46:41.107159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676610+0000","last_change":"2024-03-10T03:49:29.226991+0000","last_active":"2024-03-10T03:50:08.676610+0000","last_peered":"2024-03-10T03:50:08.676610+0000","last_clean":"2024-03-10T03:50:08.676610+0000","last_became_active":"2024-03-10T03:49:29.226687+0000","last_became_peered":"2024-03-10T03:49:29.226687+0000","last_unstale":"2024-03-10T03:50:08.676610+0000","last_undegraded":"2024-03-10T03:50:08.676610+0000","last_fullsized":"2024-03-10T03:50:08.676610+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:05:18.531132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676681+0000","last_change":"2024-03-10T03:49:25.207650+0000","last_active":"2024-03-10T03:50:08.676681+0000","last_peered":"2024-03-10T03:50:08.676681+0000","last_clean":"2024-03-10T03:50:08.676681+0000","last_became_active":"2024-03-10T03:49:25.207012+0000","last_became_peered":"2024-03-10T03:49:25.207012+0000","last_unstale":"2024-03-10T03:50:08.676681+0000","last_undegraded":"2024-03-10T03:50:08.676681+0000","last_fullsized":"2024-03-10T03:50:08.676681+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:09:03.478081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"69'23","reported_seq":67,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768262+0000","last_change":"2024-03-10T03:49:27.214213+0000","last_active":"2024-03-10T03:50:08.768262+0000","last_peered":"2024-03-10T03:50:08.768262+0000","last_clean":"2024-03-10T03:50:08.768262+0000","last_became_active":"2024-03-10T03:49:27.213911+0000","last_became_peered":"2024-03-10T03:49:27.213911+0000","last_unstale":"2024-03-10T03:50:08.768262+0000","last_undegraded":"2024-03-10T03:50:08.768262+0000","last_fullsized":"2024-03-10T03:50:08.768262+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:17:26.503750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676354+0000","last_change":"2024-03-10T03:49:31.239545+0000","last_active":"2024-03-10T03:50:08.676354+0000","last_peered":"2024-03-10T03:50:08.676354+0000","last_clean":"2024-03-10T03:50:08.676354+0000","last_became_active":"2024-03-10T03:49:31.239233+0000","last_became_peered":"2024-03-10T03:49:31.239233+0000","last_unstale":"2024-03-10T03:50:08.676354+0000","last_undegraded":"2024-03-10T03:50:08.676354+0000","last_fullsized":"2024-03-10T03:50:08.676354+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:10:12.278049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557914+0000","last_change":"2024-03-10T03:49:29.227820+0000","last_active":"2024-03-10T03:50:08.557914+0000","last_peered":"2024-03-10T03:50:08.557914+0000","last_clean":"2024-03-10T03:50:08.557914+0000","last_became_active":"2024-03-10T03:49:29.227341+0000","last_became_peered":"2024-03-10T03:49:29.227341+0000","last_unstale":"2024-03-10T03:50:08.557914+0000","last_undegraded":"2024-03-10T03:50:08.557914+0000","last_fullsized":"2024-03-10T03:50:08.557914+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:25:02.019917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676638+0000","last_change":"2024-03-10T03:49:25.208025+0000","last_active":"2024-03-10T03:50:08.676638+0000","last_peered":"2024-03-10T03:50:08.676638+0000","last_clean":"2024-03-10T03:50:08.676638+0000","last_became_active":"2024-03-10T03:49:25.207633+0000","last_became_peered":"2024-03-10T03:49:25.207633+0000","last_unstale":"2024-03-10T03:50:08.676638+0000","last_undegraded":"2024-03-10T03:50:08.676638+0000","last_fullsized":"2024-03-10T03:50:08.676638+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T14:29:10.525250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"69'11","reported_seq":47,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768337+0000","last_change":"2024-03-10T03:49:27.214164+0000","last_active":"2024-03-10T03:50:08.768337+0000","last_peered":"2024-03-10T03:50:08.768337+0000","last_clean":"2024-03-10T03:50:08.768337+0000","last_became_active":"2024-03-10T03:49:27.213892+0000","last_became_peered":"2024-03-10T03:49:27.213892+0000","last_unstale":"2024-03-10T03:50:08.768337+0000","last_undegraded":"2024-03-10T03:50:08.768337+0000","last_fullsized":"2024-03-10T03:50:08.768337+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:55:10.609679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768772+0000","last_change":"2024-03-10T03:49:31.303196+0000","last_active":"2024-03-10T03:50:08.768772+0000","last_peered":"2024-03-10T03:50:08.768772+0000","last_clean":"2024-03-10T03:50:08.768772+0000","last_became_active":"2024-03-10T03:49:31.302859+0000","last_became_peered":"2024-03-10T03:49:31.302859+0000","last_unstale":"2024-03-10T03:50:08.768772+0000","last_undegraded":"2024-03-10T03:50:08.768772+0000","last_fullsized":"2024-03-10T03:50:08.768772+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:17:17.940789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557371+0000","last_change":"2024-03-10T03:49:29.225990+0000","last_active":"2024-03-10T03:50:08.557371+0000","last_peered":"2024-03-10T03:50:08.557371+0000","last_clean":"2024-03-10T03:50:08.557371+0000","last_became_active":"2024-03-10T03:49:29.225657+0000","last_became_peered":"2024-03-10T03:49:29.225657+0000","last_unstale":"2024-03-10T03:50:08.557371+0000","last_undegraded":"2024-03-10T03:50:08.557371+0000","last_fullsized":"2024-03-10T03:50:08.557371+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:36:36.627070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557033+0000","last_change":"2024-03-10T03:49:25.207095+0000","last_active":"2024-03-10T03:50:08.557033+0000","last_peered":"2024-03-10T03:50:08.557033+0000","last_clean":"2024-03-10T03:50:08.557033+0000","last_became_active":"2024-03-10T03:49:25.206787+0000","last_became_peered":"2024-03-10T03:49:25.206787+0000","last_unstale":"2024-03-10T03:50:08.557033+0000","last_undegraded":"2024-03-10T03:50:08.557033+0000","last_fullsized":"2024-03-10T03:50:08.557033+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:05:36.111051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"69'15","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.686140+0000","last_change":"2024-03-10T03:49:27.217917+0000","last_active":"2024-03-10T03:50:08.686140+0000","last_peered":"2024-03-10T03:50:08.686140+0000","last_clean":"2024-03-10T03:50:08.686140+0000","last_became_active":"2024-03-10T03:49:27.217616+0000","last_became_peered":"2024-03-10T03:49:27.217616+0000","last_unstale":"2024-03-10T03:50:08.686140+0000","last_undegraded":"2024-03-10T03:50:08.686140+0000","last_fullsized":"2024-03-10T03:50:08.686140+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T13:04:45.259002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768720+0000","last_change":"2024-03-10T03:49:31.303033+0000","last_active":"2024-03-10T03:50:08.768720+0000","last_peered":"2024-03-10T03:50:08.768720+0000","last_clean":"2024-03-10T03:50:08.768720+0000","last_became_active":"2024-03-10T03:49:31.302693+0000","last_became_peered":"2024-03-10T03:49:31.302693+0000","last_unstale":"2024-03-10T03:50:08.768720+0000","last_undegraded":"2024-03-10T03:50:08.768720+0000","last_fullsized":"2024-03-10T03:50:08.768720+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:57:37.472778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768195+0000","last_change":"2024-03-10T03:49:29.220278+0000","last_active":"2024-03-10T03:50:08.768195+0000","last_peered":"2024-03-10T03:50:08.768195+0000","last_clean":"2024-03-10T03:50:08.768195+0000","last_became_active":"2024-03-10T03:49:29.219542+0000","last_became_peered":"2024-03-10T03:49:29.219542+0000","last_unstale":"2024-03-10T03:50:08.768195+0000","last_undegraded":"2024-03-10T03:50:08.768195+0000","last_fullsized":"2024-03-10T03:50:08.768195+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:10:23.476404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768286+0000","last_change":"2024-03-10T03:49:25.209244+0000","last_active":"2024-03-10T03:50:08.768286+0000","last_peered":"2024-03-10T03:50:08.768286+0000","last_clean":"2024-03-10T03:50:08.768286+0000","last_became_active":"2024-03-10T03:49:25.208976+0000","last_became_peered":"2024-03-10T03:49:25.208976+0000","last_unstale":"2024-03-10T03:50:08.768286+0000","last_undegraded":"2024-03-10T03:50:08.768286+0000","last_fullsized":"2024-03-10T03:50:08.768286+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:55:03.017441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"69'18","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768403+0000","last_change":"2024-03-10T03:49:27.221148+0000","last_active":"2024-03-10T03:50:08.768403+0000","last_peered":"2024-03-10T03:50:08.768403+0000","last_clean":"2024-03-10T03:50:08.768403+0000","last_became_active":"2024-03-10T03:49:27.220793+0000","last_became_peered":"2024-03-10T03:49:27.220793+0000","last_unstale":"2024-03-10T03:50:08.768403+0000","last_undegraded":"2024-03-10T03:50:08.768403+0000","last_fullsized":"2024-03-10T03:50:08.768403+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:08:25.158012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"69'1","reported_seq":19,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558523+0000","last_change":"2024-03-10T03:49:31.234581+0000","last_active":"2024-03-10T03:50:08.558523+0000","last_peered":"2024-03-10T03:50:08.558523+0000","last_clean":"2024-03-10T03:50:08.558523+0000","last_became_active":"2024-03-10T03:49:31.234331+0000","last_became_peered":"2024-03-10T03:49:31.234331+0000","last_unstale":"2024-03-10T03:50:08.558523+0000","last_undegraded":"2024-03-10T03:50:08.558523+0000","last_fullsized":"2024-03-10T03:50:08.558523+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T14:14:57.766139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"70'11","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:50.272678+0000","last_change":"2024-03-10T03:49:29.223130+0000","last_active":"2024-03-10T03:50:50.272678+0000","last_peered":"2024-03-10T03:50:50.272678+0000","last_clean":"2024-03-10T03:50:50.272678+0000","last_became_active":"2024-03-10T03:49:29.222801+0000","last_became_peered":"2024-03-10T03:49:29.222801+0000","last_unstale":"2024-03-10T03:50:50.272678+0000","last_undegraded":"2024-03-10T03:50:50.272678+0000","last_fullsized":"2024-03-10T03:50:50.272678+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:58:55.411306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"62'1","reported_seq":30,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558151+0000","last_change":"2024-03-10T03:49:25.209690+0000","last_active":"2024-03-10T03:50:08.558151+0000","last_peered":"2024-03-10T03:50:08.558151+0000","last_clean":"2024-03-10T03:50:08.558151+0000","last_became_active":"2024-03-10T03:49:25.209411+0000","last_became_peered":"2024-03-10T03:49:25.209411+0000","last_unstale":"2024-03-10T03:50:08.558151+0000","last_undegraded":"2024-03-10T03:50:08.558151+0000","last_fullsized":"2024-03-10T03:50:08.558151+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T15:07:12.056710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"69'16","reported_seq":52,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.556995+0000","last_change":"2024-03-10T03:49:27.226494+0000","last_active":"2024-03-10T03:50:08.556995+0000","last_peered":"2024-03-10T03:50:08.556995+0000","last_clean":"2024-03-10T03:50:08.556995+0000","last_became_active":"2024-03-10T03:49:27.226193+0000","last_became_peered":"2024-03-10T03:49:27.226193+0000","last_unstale":"2024-03-10T03:50:08.556995+0000","last_undegraded":"2024-03-10T03:50:08.556995+0000","last_fullsized":"2024-03-10T03:50:08.556995+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T13:58:19.445874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557533+0000","last_change":"2024-03-10T03:49:31.231756+0000","last_active":"2024-03-10T03:50:08.557533+0000","last_peered":"2024-03-10T03:50:08.557533+0000","last_clean":"2024-03-10T03:50:08.557533+0000","last_became_active":"2024-03-10T03:49:31.231396+0000","last_became_peered":"2024-03-10T03:49:31.231396+0000","last_unstale":"2024-03-10T03:50:08.557533+0000","last_undegraded":"2024-03-10T03:50:08.557533+0000","last_fullsized":"2024-03-10T03:50:08.557533+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:07:37.738114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557324+0000","last_change":"2024-03-10T03:49:29.221814+0000","last_active":"2024-03-10T03:50:08.557324+0000","last_peered":"2024-03-10T03:50:08.557324+0000","last_clean":"2024-03-10T03:50:08.557324+0000","last_became_active":"2024-03-10T03:49:29.221518+0000","last_became_peered":"2024-03-10T03:49:29.221518+0000","last_unstale":"2024-03-10T03:50:08.557324+0000","last_undegraded":"2024-03-10T03:50:08.557324+0000","last_fullsized":"2024-03-10T03:50:08.557324+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:41:24.206929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767256+0000","last_change":"2024-03-10T03:49:25.214747+0000","last_active":"2024-03-10T03:50:08.767256+0000","last_peered":"2024-03-10T03:50:08.767256+0000","last_clean":"2024-03-10T03:50:08.767256+0000","last_became_active":"2024-03-10T03:49:25.214471+0000","last_became_peered":"2024-03-10T03:49:25.214471+0000","last_unstale":"2024-03-10T03:50:08.767256+0000","last_undegraded":"2024-03-10T03:50:08.767256+0000","last_fullsized":"2024-03-10T03:50:08.767256+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T15:21:53.263960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"69'17","reported_seq":56,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767574+0000","last_change":"2024-03-10T03:49:27.220041+0000","last_active":"2024-03-10T03:50:08.767574+0000","last_peered":"2024-03-10T03:50:08.767574+0000","last_clean":"2024-03-10T03:50:08.767574+0000","last_became_active":"2024-03-10T03:49:27.219709+0000","last_became_peered":"2024-03-10T03:49:27.219709+0000","last_unstale":"2024-03-10T03:50:08.767574+0000","last_undegraded":"2024-03-10T03:50:08.767574+0000","last_fullsized":"2024-03-10T03:50:08.767574+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T15:22:04.462073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557822+0000","last_change":"2024-03-10T03:49:31.237845+0000","last_active":"2024-03-10T03:50:08.557822+0000","last_peered":"2024-03-10T03:50:08.557822+0000","last_clean":"2024-03-10T03:50:08.557822+0000","last_became_active":"2024-03-10T03:49:31.237518+0000","last_became_peered":"2024-03-10T03:49:31.237518+0000","last_unstale":"2024-03-10T03:50:08.557822+0000","last_undegraded":"2024-03-10T03:50:08.557822+0000","last_fullsized":"2024-03-10T03:50:08.557822+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T03:51:07.465866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767249+0000","last_change":"2024-03-10T03:49:29.229795+0000","last_active":"2024-03-10T03:50:08.767249+0000","last_peered":"2024-03-10T03:50:08.767249+0000","last_clean":"2024-03-10T03:50:08.767249+0000","last_became_active":"2024-03-10T03:49:29.229496+0000","last_became_peered":"2024-03-10T03:49:29.229496+0000","last_unstale":"2024-03-10T03:50:08.767249+0000","last_undegraded":"2024-03-10T03:50:08.767249+0000","last_fullsized":"2024-03-10T03:50:08.767249+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:38:17.789071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557789+0000","last_change":"2024-03-10T03:49:25.216659+0000","last_active":"2024-03-10T03:50:08.557789+0000","last_peered":"2024-03-10T03:50:08.557789+0000","last_clean":"2024-03-10T03:50:08.557789+0000","last_became_active":"2024-03-10T03:49:25.216334+0000","last_became_peered":"2024-03-10T03:49:25.216334+0000","last_unstale":"2024-03-10T03:50:08.557789+0000","last_undegraded":"2024-03-10T03:50:08.557789+0000","last_fullsized":"2024-03-10T03:50:08.557789+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T13:02:37.295580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"69'4","reported_seq":31,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767328+0000","last_change":"2024-03-10T03:49:27.220567+0000","last_active":"2024-03-10T03:50:08.767328+0000","last_peered":"2024-03-10T03:50:08.767328+0000","last_clean":"2024-03-10T03:50:08.767328+0000","last_became_active":"2024-03-10T03:49:27.220255+0000","last_became_peered":"2024-03-10T03:49:27.220255+0000","last_unstale":"2024-03-10T03:50:08.767328+0000","last_undegraded":"2024-03-10T03:50:08.767328+0000","last_fullsized":"2024-03-10T03:50:08.767328+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T13:51:27.826682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767720+0000","last_change":"2024-03-10T03:49:31.304998+0000","last_active":"2024-03-10T03:50:08.767720+0000","last_peered":"2024-03-10T03:50:08.767720+0000","last_clean":"2024-03-10T03:50:08.767720+0000","last_became_active":"2024-03-10T03:49:31.304515+0000","last_became_peered":"2024-03-10T03:49:31.304515+0000","last_unstale":"2024-03-10T03:50:08.767720+0000","last_undegraded":"2024-03-10T03:50:08.767720+0000","last_fullsized":"2024-03-10T03:50:08.767720+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:01:01.378335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558567+0000","last_change":"2024-03-10T03:49:29.228581+0000","last_active":"2024-03-10T03:50:08.558567+0000","last_peered":"2024-03-10T03:50:08.558567+0000","last_clean":"2024-03-10T03:50:08.558567+0000","last_became_active":"2024-03-10T03:49:29.228279+0000","last_became_peered":"2024-03-10T03:49:29.228279+0000","last_unstale":"2024-03-10T03:50:08.558567+0000","last_undegraded":"2024-03-10T03:50:08.558567+0000","last_fullsized":"2024-03-10T03:50:08.558567+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:21:49.987568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676863+0000","last_change":"2024-03-10T03:49:25.204605+0000","last_active":"2024-03-10T03:50:08.676863+0000","last_peered":"2024-03-10T03:50:08.676863+0000","last_clean":"2024-03-10T03:50:08.676863+0000","last_became_active":"2024-03-10T03:49:25.204342+0000","last_became_peered":"2024-03-10T03:49:25.204342+0000","last_unstale":"2024-03-10T03:50:08.676863+0000","last_undegraded":"2024-03-10T03:50:08.676863+0000","last_fullsized":"2024-03-10T03:50:08.676863+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:08:03.853948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"69'17","reported_seq":56,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767552+0000","last_change":"2024-03-10T03:49:27.224564+0000","last_active":"2024-03-10T03:50:08.767552+0000","last_peered":"2024-03-10T03:50:08.767552+0000","last_clean":"2024-03-10T03:50:08.767552+0000","last_became_active":"2024-03-10T03:49:27.224268+0000","last_became_peered":"2024-03-10T03:49:27.224268+0000","last_unstale":"2024-03-10T03:50:08.767552+0000","last_undegraded":"2024-03-10T03:50:08.767552+0000","last_fullsized":"2024-03-10T03:50:08.767552+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:06:09.308204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.555904+0000","last_change":"2024-03-10T03:49:31.232315+0000","last_active":"2024-03-10T03:50:08.555904+0000","last_peered":"2024-03-10T03:50:08.555904+0000","last_clean":"2024-03-10T03:50:08.555904+0000","last_became_active":"2024-03-10T03:49:31.231981+0000","last_became_peered":"2024-03-10T03:49:31.231981+0000","last_unstale":"2024-03-10T03:50:08.555904+0000","last_undegraded":"2024-03-10T03:50:08.555904+0000","last_fullsized":"2024-03-10T03:50:08.555904+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:34:16.673037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767961+0000","last_change":"2024-03-10T03:49:29.222667+0000","last_active":"2024-03-10T03:50:08.767961+0000","last_peered":"2024-03-10T03:50:08.767961+0000","last_clean":"2024-03-10T03:50:08.767961+0000","last_became_active":"2024-03-10T03:49:29.222378+0000","last_became_peered":"2024-03-10T03:49:29.222378+0000","last_unstale":"2024-03-10T03:50:08.767961+0000","last_undegraded":"2024-03-10T03:50:08.767961+0000","last_fullsized":"2024-03-10T03:50:08.767961+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:12:44.452463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558651+0000","last_change":"2024-03-10T03:49:25.214331+0000","last_active":"2024-03-10T03:50:08.558651+0000","last_peered":"2024-03-10T03:50:08.558651+0000","last_clean":"2024-03-10T03:50:08.558651+0000","last_became_active":"2024-03-10T03:49:25.214039+0000","last_became_peered":"2024-03-10T03:49:25.214039+0000","last_unstale":"2024-03-10T03:50:08.558651+0000","last_undegraded":"2024-03-10T03:50:08.558651+0000","last_fullsized":"2024-03-10T03:50:08.558651+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T13:20:23.361735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"69'9","reported_seq":41,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.559840+0000","last_change":"2024-03-10T03:49:27.213754+0000","last_active":"2024-03-10T03:50:08.559840+0000","last_peered":"2024-03-10T03:50:08.559840+0000","last_clean":"2024-03-10T03:50:08.559840+0000","last_became_active":"2024-03-10T03:49:27.213363+0000","last_became_peered":"2024-03-10T03:49:27.213363+0000","last_unstale":"2024-03-10T03:50:08.559840+0000","last_undegraded":"2024-03-10T03:50:08.559840+0000","last_fullsized":"2024-03-10T03:50:08.559840+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:05:49.339755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767363+0000","last_change":"2024-03-10T03:49:31.241899+0000","last_active":"2024-03-10T03:50:08.767363+0000","last_peered":"2024-03-10T03:50:08.767363+0000","last_clean":"2024-03-10T03:50:08.767363+0000","last_became_active":"2024-03-10T03:49:31.241459+0000","last_became_peered":"2024-03-10T03:49:31.241459+0000","last_unstale":"2024-03-10T03:50:08.767363+0000","last_undegraded":"2024-03-10T03:50:08.767363+0000","last_fullsized":"2024-03-10T03:50:08.767363+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:52:23.376147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.559599+0000","last_change":"2024-03-10T03:49:29.224799+0000","last_active":"2024-03-10T03:50:08.559599+0000","last_peered":"2024-03-10T03:50:08.559599+0000","last_clean":"2024-03-10T03:50:08.559599+0000","last_became_active":"2024-03-10T03:49:29.224448+0000","last_became_peered":"2024-03-10T03:49:29.224448+0000","last_unstale":"2024-03-10T03:50:08.559599+0000","last_undegraded":"2024-03-10T03:50:08.559599+0000","last_fullsized":"2024-03-10T03:50:08.559599+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T13:35:21.264435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768457+0000","last_change":"2024-03-10T03:49:25.211384+0000","last_active":"2024-03-10T03:50:08.768457+0000","last_peered":"2024-03-10T03:50:08.768457+0000","last_clean":"2024-03-10T03:50:08.768457+0000","last_became_active":"2024-03-10T03:49:25.211105+0000","last_became_peered":"2024-03-10T03:49:25.211105+0000","last_unstale":"2024-03-10T03:50:08.768457+0000","last_undegraded":"2024-03-10T03:50:08.768457+0000","last_fullsized":"2024-03-10T03:50:08.768457+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:41:09.133320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"69'11","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768518+0000","last_change":"2024-03-10T03:49:27.208982+0000","last_active":"2024-03-10T03:50:08.768518+0000","last_peered":"2024-03-10T03:50:08.768518+0000","last_clean":"2024-03-10T03:50:08.768518+0000","last_became_active":"2024-03-10T03:49:27.208682+0000","last_became_peered":"2024-03-10T03:49:27.208682+0000","last_unstale":"2024-03-10T03:50:08.768518+0000","last_undegraded":"2024-03-10T03:50:08.768518+0000","last_fullsized":"2024-03-10T03:50:08.768518+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:15:07.281839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.556670+0000","last_change":"2024-03-10T03:49:31.242771+0000","last_active":"2024-03-10T03:50:08.556670+0000","last_peered":"2024-03-10T03:50:08.556670+0000","last_clean":"2024-03-10T03:50:08.556670+0000","last_became_active":"2024-03-10T03:49:31.242375+0000","last_became_peered":"2024-03-10T03:49:31.242375+0000","last_unstale":"2024-03-10T03:50:08.556670+0000","last_undegraded":"2024-03-10T03:50:08.556670+0000","last_fullsized":"2024-03-10T03:50:08.556670+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T14:51:17.780789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"70'11","reported_seq":51,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:50.272685+0000","last_change":"2024-03-10T03:49:29.225588+0000","last_active":"2024-03-10T03:50:50.272685+0000","last_peered":"2024-03-10T03:50:50.272685+0000","last_clean":"2024-03-10T03:50:50.272685+0000","last_became_active":"2024-03-10T03:49:29.225281+0000","last_became_peered":"2024-03-10T03:49:29.225281+0000","last_unstale":"2024-03-10T03:50:50.272685+0000","last_undegraded":"2024-03-10T03:50:50.272685+0000","last_fullsized":"2024-03-10T03:50:50.272685+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T13:00:58.748745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557993+0000","last_change":"2024-03-10T03:49:25.210071+0000","last_active":"2024-03-10T03:50:08.557993+0000","last_peered":"2024-03-10T03:50:08.557993+0000","last_clean":"2024-03-10T03:50:08.557993+0000","last_became_active":"2024-03-10T03:49:25.209806+0000","last_became_peered":"2024-03-10T03:49:25.209806+0000","last_unstale":"2024-03-10T03:50:08.557993+0000","last_undegraded":"2024-03-10T03:50:08.557993+0000","last_fullsized":"2024-03-10T03:50:08.557993+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:05:18.614686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"69'22","reported_seq":64,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768001+0000","last_change":"2024-03-10T03:49:27.227462+0000","last_active":"2024-03-10T03:50:08.768001+0000","last_peered":"2024-03-10T03:50:08.768001+0000","last_clean":"2024-03-10T03:50:08.768001+0000","last_became_active":"2024-03-10T03:49:27.227085+0000","last_became_peered":"2024-03-10T03:49:27.227085+0000","last_unstale":"2024-03-10T03:50:08.768001+0000","last_undegraded":"2024-03-10T03:50:08.768001+0000","last_fullsized":"2024-03-10T03:50:08.768001+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:42:03.647651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768686+0000","last_change":"2024-03-10T03:49:31.237861+0000","last_active":"2024-03-10T03:50:08.768686+0000","last_peered":"2024-03-10T03:50:08.768686+0000","last_clean":"2024-03-10T03:50:08.768686+0000","last_became_active":"2024-03-10T03:49:31.237584+0000","last_became_peered":"2024-03-10T03:49:31.237584+0000","last_unstale":"2024-03-10T03:50:08.768686+0000","last_undegraded":"2024-03-10T03:50:08.768686+0000","last_fullsized":"2024-03-10T03:50:08.768686+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:12:19.353438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"70'11","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:50.272724+0000","last_change":"2024-03-10T03:49:29.221165+0000","last_active":"2024-03-10T03:50:50.272724+0000","last_peered":"2024-03-10T03:50:50.272724+0000","last_clean":"2024-03-10T03:50:50.272724+0000","last_became_active":"2024-03-10T03:49:29.220865+0000","last_became_peered":"2024-03-10T03:49:29.220865+0000","last_unstale":"2024-03-10T03:50:50.272724+0000","last_undegraded":"2024-03-10T03:50:50.272724+0000","last_fullsized":"2024-03-10T03:50:50.272724+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T15:28:08.462333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.556275+0000","last_change":"2024-03-10T03:49:25.216824+0000","last_active":"2024-03-10T03:50:08.556275+0000","last_peered":"2024-03-10T03:50:08.556275+0000","last_clean":"2024-03-10T03:50:08.556275+0000","last_became_active":"2024-03-10T03:49:25.216569+0000","last_became_peered":"2024-03-10T03:49:25.216569+0000","last_unstale":"2024-03-10T03:50:08.556275+0000","last_undegraded":"2024-03-10T03:50:08.556275+0000","last_fullsized":"2024-03-10T03:50:08.556275+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:23:48.509540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"69'15","reported_seq":53,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676967+0000","last_change":"2024-03-10T03:49:27.225582+0000","last_active":"2024-03-10T03:50:08.676967+0000","last_peered":"2024-03-10T03:50:08.676967+0000","last_clean":"2024-03-10T03:50:08.676967+0000","last_became_active":"2024-03-10T03:49:27.225278+0000","last_became_peered":"2024-03-10T03:49:27.225278+0000","last_unstale":"2024-03-10T03:50:08.676967+0000","last_undegraded":"2024-03-10T03:50:08.676967+0000","last_fullsized":"2024-03-10T03:50:08.676967+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:57:10.878877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.686483+0000","last_change":"2024-03-10T03:49:31.234849+0000","last_active":"2024-03-10T03:50:08.686483+0000","last_peered":"2024-03-10T03:50:08.686483+0000","last_clean":"2024-03-10T03:50:08.686483+0000","last_became_active":"2024-03-10T03:49:31.234438+0000","last_became_peered":"2024-03-10T03:49:31.234438+0000","last_unstale":"2024-03-10T03:50:08.686483+0000","last_undegraded":"2024-03-10T03:50:08.686483+0000","last_fullsized":"2024-03-10T03:50:08.686483+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T14:49:05.413593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768691+0000","last_change":"2024-03-10T03:49:29.227165+0000","last_active":"2024-03-10T03:50:08.768691+0000","last_peered":"2024-03-10T03:50:08.768691+0000","last_clean":"2024-03-10T03:50:08.768691+0000","last_became_active":"2024-03-10T03:49:29.226840+0000","last_became_peered":"2024-03-10T03:49:29.226840+0000","last_unstale":"2024-03-10T03:50:08.768691+0000","last_undegraded":"2024-03-10T03:50:08.768691+0000","last_fullsized":"2024-03-10T03:50:08.768691+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T13:05:47.034971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"62'1","reported_seq":37,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558012+0000","last_change":"2024-03-10T03:49:25.212208+0000","last_active":"2024-03-10T03:50:08.558012+0000","last_peered":"2024-03-10T03:50:08.558012+0000","last_clean":"2024-03-10T03:50:08.558012+0000","last_became_active":"2024-03-10T03:49:25.211908+0000","last_became_peered":"2024-03-10T03:49:25.211908+0000","last_unstale":"2024-03-10T03:50:08.558012+0000","last_undegraded":"2024-03-10T03:50:08.558012+0000","last_fullsized":"2024-03-10T03:50:08.558012+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:46:41.351712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"69'21","reported_seq":65,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557137+0000","last_change":"2024-03-10T03:49:27.225782+0000","last_active":"2024-03-10T03:50:08.557137+0000","last_peered":"2024-03-10T03:50:08.557137+0000","last_clean":"2024-03-10T03:50:08.557137+0000","last_became_active":"2024-03-10T03:49:27.225487+0000","last_became_peered":"2024-03-10T03:49:27.225487+0000","last_unstale":"2024-03-10T03:50:08.557137+0000","last_undegraded":"2024-03-10T03:50:08.557137+0000","last_fullsized":"2024-03-10T03:50:08.557137+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T13:23:13.259430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558548+0000","last_change":"2024-03-10T03:49:31.305179+0000","last_active":"2024-03-10T03:50:08.558548+0000","last_peered":"2024-03-10T03:50:08.558548+0000","last_clean":"2024-03-10T03:50:08.558548+0000","last_became_active":"2024-03-10T03:49:31.304779+0000","last_became_peered":"2024-03-10T03:49:31.304779+0000","last_unstale":"2024-03-10T03:50:08.558548+0000","last_undegraded":"2024-03-10T03:50:08.558548+0000","last_fullsized":"2024-03-10T03:50:08.558548+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:16:31.779939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676909+0000","last_change":"2024-03-10T03:49:29.222395+0000","last_active":"2024-03-10T03:50:08.676909+0000","last_peered":"2024-03-10T03:50:08.676909+0000","last_clean":"2024-03-10T03:50:08.676909+0000","last_became_active":"2024-03-10T03:49:29.222089+0000","last_became_peered":"2024-03-10T03:49:29.222089+0000","last_unstale":"2024-03-10T03:50:08.676909+0000","last_undegraded":"2024-03-10T03:50:08.676909+0000","last_fullsized":"2024-03-10T03:50:08.676909+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T15:26:51.165830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767194+0000","last_change":"2024-03-10T03:49:25.214883+0000","last_active":"2024-03-10T03:50:08.767194+0000","last_peered":"2024-03-10T03:50:08.767194+0000","last_clean":"2024-03-10T03:50:08.767194+0000","last_became_active":"2024-03-10T03:49:25.214593+0000","last_became_peered":"2024-03-10T03:49:25.214593+0000","last_unstale":"2024-03-10T03:50:08.767194+0000","last_undegraded":"2024-03-10T03:50:08.767194+0000","last_fullsized":"2024-03-10T03:50:08.767194+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:27:40.787890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"69'12","reported_seq":46,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768783+0000","last_change":"2024-03-10T03:49:27.226530+0000","last_active":"2024-03-10T03:50:08.768783+0000","last_peered":"2024-03-10T03:50:08.768783+0000","last_clean":"2024-03-10T03:50:08.768783+0000","last_became_active":"2024-03-10T03:49:27.226227+0000","last_became_peered":"2024-03-10T03:49:27.226227+0000","last_unstale":"2024-03-10T03:50:08.768783+0000","last_undegraded":"2024-03-10T03:50:08.768783+0000","last_fullsized":"2024-03-10T03:50:08.768783+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:52:29.571582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676818+0000","last_change":"2024-03-10T03:49:31.302968+0000","last_active":"2024-03-10T03:50:08.676818+0000","last_peered":"2024-03-10T03:50:08.676818+0000","last_clean":"2024-03-10T03:50:08.676818+0000","last_became_active":"2024-03-10T03:49:31.302495+0000","last_became_peered":"2024-03-10T03:49:31.302495+0000","last_unstale":"2024-03-10T03:50:08.676818+0000","last_undegraded":"2024-03-10T03:50:08.676818+0000","last_fullsized":"2024-03-10T03:50:08.676818+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:59:51.712808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"62'2","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.559132+0000","last_change":"2024-03-10T03:49:25.213952+0000","last_active":"2024-03-10T03:50:08.559132+0000","last_peered":"2024-03-10T03:50:08.559132+0000","last_clean":"2024-03-10T03:50:08.559132+0000","last_became_active":"2024-03-10T03:49:25.213652+0000","last_became_peered":"2024-03-10T03:49:25.213652+0000","last_unstale":"2024-03-10T03:50:08.559132+0000","last_undegraded":"2024-03-10T03:50:08.559132+0000","last_fullsized":"2024-03-10T03:50:08.559132+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T14:20:41.107876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"69'27","reported_seq":74,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676379+0000","last_change":"2024-03-10T03:49:27.226178+0000","last_active":"2024-03-10T03:50:08.676379+0000","last_peered":"2024-03-10T03:50:08.676379+0000","last_clean":"2024-03-10T03:50:08.676379+0000","last_became_active":"2024-03-10T03:49:27.225871+0000","last_became_peered":"2024-03-10T03:49:27.225871+0000","last_unstale":"2024-03-10T03:50:08.676379+0000","last_undegraded":"2024-03-10T03:50:08.676379+0000","last_fullsized":"2024-03-10T03:50:08.676379+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T15:01:50.869781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"70'11","reported_seq":51,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:50.272871+0000","last_change":"2024-03-10T03:49:29.230442+0000","last_active":"2024-03-10T03:50:50.272871+0000","last_peered":"2024-03-10T03:50:50.272871+0000","last_clean":"2024-03-10T03:50:50.272871+0000","last_became_active":"2024-03-10T03:49:29.230156+0000","last_became_peered":"2024-03-10T03:49:29.230156+0000","last_unstale":"2024-03-10T03:50:50.272871+0000","last_undegraded":"2024-03-10T03:50:50.272871+0000","last_fullsized":"2024-03-10T03:50:50.272871+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:36:09.029657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767865+0000","last_change":"2024-03-10T03:49:31.238755+0000","last_active":"2024-03-10T03:50:08.767865+0000","last_peered":"2024-03-10T03:50:08.767865+0000","last_clean":"2024-03-10T03:50:08.767865+0000","last_became_active":"2024-03-10T03:49:31.237221+0000","last_became_peered":"2024-03-10T03:49:31.237221+0000","last_unstale":"2024-03-10T03:50:08.767865+0000","last_undegraded":"2024-03-10T03:50:08.767865+0000","last_fullsized":"2024-03-10T03:50:08.767865+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:49:11.201929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558286+0000","last_change":"2024-03-10T03:49:25.210399+0000","last_active":"2024-03-10T03:50:08.558286+0000","last_peered":"2024-03-10T03:50:08.558286+0000","last_clean":"2024-03-10T03:50:08.558286+0000","last_became_active":"2024-03-10T03:49:25.209796+0000","last_became_peered":"2024-03-10T03:49:25.209796+0000","last_unstale":"2024-03-10T03:50:08.558286+0000","last_undegraded":"2024-03-10T03:50:08.558286+0000","last_fullsized":"2024-03-10T03:50:08.558286+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:14:18.333004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"69'24","reported_seq":72,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768094+0000","last_change":"2024-03-10T03:49:27.217870+0000","last_active":"2024-03-10T03:50:08.768094+0000","last_peered":"2024-03-10T03:50:08.768094+0000","last_clean":"2024-03-10T03:50:08.768094+0000","last_became_active":"2024-03-10T03:49:27.217694+0000","last_became_peered":"2024-03-10T03:49:27.217694+0000","last_unstale":"2024-03-10T03:50:08.768094+0000","last_undegraded":"2024-03-10T03:50:08.768094+0000","last_fullsized":"2024-03-10T03:50:08.768094+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:05:26.866353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.685681+0000","last_change":"2024-03-10T03:49:29.221281+0000","last_active":"2024-03-10T03:50:08.685681+0000","last_peered":"2024-03-10T03:50:08.685681+0000","last_clean":"2024-03-10T03:50:08.685681+0000","last_became_active":"2024-03-10T03:49:29.220944+0000","last_became_peered":"2024-03-10T03:49:29.220944+0000","last_unstale":"2024-03-10T03:50:08.685681+0000","last_undegraded":"2024-03-10T03:50:08.685681+0000","last_fullsized":"2024-03-10T03:50:08.685681+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:58:44.147234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557618+0000","last_change":"2024-03-10T03:49:31.240182+0000","last_active":"2024-03-10T03:50:08.557618+0000","last_peered":"2024-03-10T03:50:08.557618+0000","last_clean":"2024-03-10T03:50:08.557618+0000","last_became_active":"2024-03-10T03:49:31.239844+0000","last_became_peered":"2024-03-10T03:49:31.239844+0000","last_unstale":"2024-03-10T03:50:08.557618+0000","last_undegraded":"2024-03-10T03:50:08.557618+0000","last_fullsized":"2024-03-10T03:50:08.557618+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:03:38.448627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.686034+0000","last_change":"2024-03-10T03:49:29.226447+0000","last_active":"2024-03-10T03:50:08.686034+0000","last_peered":"2024-03-10T03:50:08.686034+0000","last_clean":"2024-03-10T03:50:08.686034+0000","last_became_active":"2024-03-10T03:49:29.226165+0000","last_became_peered":"2024-03-10T03:49:29.226165+0000","last_unstale":"2024-03-10T03:50:08.686034+0000","last_undegraded":"2024-03-10T03:50:08.686034+0000","last_fullsized":"2024-03-10T03:50:08.686034+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:08:06.601491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558861+0000","last_change":"2024-03-10T03:49:25.208976+0000","last_active":"2024-03-10T03:50:08.558861+0000","last_peered":"2024-03-10T03:50:08.558861+0000","last_clean":"2024-03-10T03:50:08.558861+0000","last_became_active":"2024-03-10T03:49:25.208386+0000","last_became_peered":"2024-03-10T03:49:25.208386+0000","last_unstale":"2024-03-10T03:50:08.558861+0000","last_undegraded":"2024-03-10T03:50:08.558861+0000","last_fullsized":"2024-03-10T03:50:08.558861+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:20:13.465647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"69'25","reported_seq":68,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767577+0000","last_change":"2024-03-10T03:49:27.225629+0000","last_active":"2024-03-10T03:50:08.767577+0000","last_peered":"2024-03-10T03:50:08.767577+0000","last_clean":"2024-03-10T03:50:08.767577+0000","last_became_active":"2024-03-10T03:49:27.225286+0000","last_became_peered":"2024-03-10T03:49:27.225286+0000","last_unstale":"2024-03-10T03:50:08.767577+0000","last_undegraded":"2024-03-10T03:50:08.767577+0000","last_fullsized":"2024-03-10T03:50:08.767577+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:04:53.452163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.556067+0000","last_change":"2024-03-10T03:49:31.233965+0000","last_active":"2024-03-10T03:50:08.556067+0000","last_peered":"2024-03-10T03:50:08.556067+0000","last_clean":"2024-03-10T03:50:08.556067+0000","last_became_active":"2024-03-10T03:49:31.233033+0000","last_became_peered":"2024-03-10T03:49:31.233033+0000","last_unstale":"2024-03-10T03:50:08.556067+0000","last_undegraded":"2024-03-10T03:50:08.556067+0000","last_fullsized":"2024-03-10T03:50:08.556067+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T13:31:56.437954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.685999+0000","last_change":"2024-03-10T03:49:29.218933+0000","last_active":"2024-03-10T03:50:08.685999+0000","last_peered":"2024-03-10T03:50:08.685999+0000","last_clean":"2024-03-10T03:50:08.685999+0000","last_became_active":"2024-03-10T03:49:29.218638+0000","last_became_peered":"2024-03-10T03:49:29.218638+0000","last_unstale":"2024-03-10T03:50:08.685999+0000","last_undegraded":"2024-03-10T03:50:08.685999+0000","last_fullsized":"2024-03-10T03:50:08.685999+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:34:56.549956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676668+0000","last_change":"2024-03-10T03:49:25.211190+0000","last_active":"2024-03-10T03:50:08.676668+0000","last_peered":"2024-03-10T03:50:08.676668+0000","last_clean":"2024-03-10T03:50:08.676668+0000","last_became_active":"2024-03-10T03:49:25.210855+0000","last_became_peered":"2024-03-10T03:49:25.210855+0000","last_unstale":"2024-03-10T03:50:08.676668+0000","last_undegraded":"2024-03-10T03:50:08.676668+0000","last_fullsized":"2024-03-10T03:50:08.676668+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:30:07.502998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"69'15","reported_seq":53,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557441+0000","last_change":"2024-03-10T03:49:27.214715+0000","last_active":"2024-03-10T03:50:08.557441+0000","last_peered":"2024-03-10T03:50:08.557441+0000","last_clean":"2024-03-10T03:50:08.557441+0000","last_became_active":"2024-03-10T03:49:27.214438+0000","last_became_peered":"2024-03-10T03:49:27.214438+0000","last_unstale":"2024-03-10T03:50:08.557441+0000","last_undegraded":"2024-03-10T03:50:08.557441+0000","last_fullsized":"2024-03-10T03:50:08.557441+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T14:19:49.905441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768156+0000","last_change":"2024-03-10T03:49:31.238061+0000","last_active":"2024-03-10T03:50:08.768156+0000","last_peered":"2024-03-10T03:50:08.768156+0000","last_clean":"2024-03-10T03:50:08.768156+0000","last_became_active":"2024-03-10T03:49:31.237686+0000","last_became_peered":"2024-03-10T03:49:31.237686+0000","last_unstale":"2024-03-10T03:50:08.768156+0000","last_undegraded":"2024-03-10T03:50:08.768156+0000","last_fullsized":"2024-03-10T03:50:08.768156+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:54:12.652235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"66'2","reported_seq":38,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676524+0000","last_change":"2024-03-10T03:49:29.223417+0000","last_active":"2024-03-10T03:50:08.676524+0000","last_peered":"2024-03-10T03:50:08.676524+0000","last_clean":"2024-03-10T03:50:08.676524+0000","last_became_active":"2024-03-10T03:49:25.204053+0000","last_became_peered":"2024-03-10T03:49:25.204053+0000","last_unstale":"2024-03-10T03:50:08.676524+0000","last_undegraded":"2024-03-10T03:50:08.676524+0000","last_fullsized":"2024-03-10T03:50:08.676524+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:44:55.447963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049149550000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768055+0000","last_change":"2024-03-10T03:49:25.214213+0000","last_active":"2024-03-10T03:50:08.768055+0000","last_peered":"2024-03-10T03:50:08.768055+0000","last_clean":"2024-03-10T03:50:08.768055+0000","last_became_active":"2024-03-10T03:49:25.213894+0000","last_became_peered":"2024-03-10T03:49:25.213894+0000","last_unstale":"2024-03-10T03:50:08.768055+0000","last_undegraded":"2024-03-10T03:50:08.768055+0000","last_fullsized":"2024-03-10T03:50:08.768055+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:12:39.847132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"70'36","reported_seq":107,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:50.272687+0000","last_change":"2024-03-10T03:49:27.216890+0000","last_active":"2024-03-10T03:50:50.272687+0000","last_peered":"2024-03-10T03:50:50.272687+0000","last_clean":"2024-03-10T03:50:50.272687+0000","last_became_active":"2024-03-10T03:49:27.216607+0000","last_became_peered":"2024-03-10T03:49:27.216607+0000","last_unstale":"2024-03-10T03:50:50.272687+0000","last_undegraded":"2024-03-10T03:50:50.272687+0000","last_fullsized":"2024-03-10T03:50:50.272687+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:40:46.914229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":60,"num_read_kb":44,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.556565+0000","last_change":"2024-03-10T03:49:29.223573+0000","last_active":"2024-03-10T03:50:08.556565+0000","last_peered":"2024-03-10T03:50:08.556565+0000","last_clean":"2024-03-10T03:50:08.556565+0000","last_became_active":"2024-03-10T03:49:29.223300+0000","last_became_peered":"2024-03-10T03:49:29.223300+0000","last_unstale":"2024-03-10T03:50:08.556565+0000","last_undegraded":"2024-03-10T03:50:08.556565+0000","last_fullsized":"2024-03-10T03:50:08.556565+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:13:35.609350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.559015+0000","last_change":"2024-03-10T03:49:31.303370+0000","last_active":"2024-03-10T03:50:08.559015+0000","last_peered":"2024-03-10T03:50:08.559015+0000","last_clean":"2024-03-10T03:50:08.559015+0000","last_became_active":"2024-03-10T03:49:31.302937+0000","last_became_peered":"2024-03-10T03:49:31.302937+0000","last_unstale":"2024-03-10T03:50:08.559015+0000","last_undegraded":"2024-03-10T03:50:08.559015+0000","last_fullsized":"2024-03-10T03:50:08.559015+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:48:59.980889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557548+0000","last_change":"2024-03-10T03:49:25.208243+0000","last_active":"2024-03-10T03:50:08.557548+0000","last_peered":"2024-03-10T03:50:08.557548+0000","last_clean":"2024-03-10T03:50:08.557548+0000","last_became_active":"2024-03-10T03:49:25.207949+0000","last_became_peered":"2024-03-10T03:49:25.207949+0000","last_unstale":"2024-03-10T03:50:08.557548+0000","last_undegraded":"2024-03-10T03:50:08.557548+0000","last_fullsized":"2024-03-10T03:50:08.557548+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T14:03:52.412562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":36,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.685882+0000","last_change":"2024-03-10T03:49:29.214944+0000","last_active":"2024-03-10T03:50:08.685882+0000","last_peered":"2024-03-10T03:50:08.685882+0000","last_clean":"2024-03-10T03:50:08.685882+0000","last_became_active":"2024-03-10T03:49:25.212958+0000","last_became_peered":"2024-03-10T03:49:25.212958+0000","last_unstale":"2024-03-10T03:50:08.685882+0000","last_undegraded":"2024-03-10T03:50:08.685882+0000","last_fullsized":"2024-03-10T03:50:08.685882+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:42:22.054366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003673029,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"69'19","reported_seq":64,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558514+0000","last_change":"2024-03-10T03:49:27.217528+0000","last_active":"2024-03-10T03:50:08.558514+0000","last_peered":"2024-03-10T03:50:08.558514+0000","last_clean":"2024-03-10T03:50:08.558514+0000","last_became_active":"2024-03-10T03:49:27.217249+0000","last_became_peered":"2024-03-10T03:49:27.217249+0000","last_unstale":"2024-03-10T03:50:08.558514+0000","last_undegraded":"2024-03-10T03:50:08.558514+0000","last_fullsized":"2024-03-10T03:50:08.558514+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:48:56.758854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.686273+0000","last_change":"2024-03-10T03:49:29.225740+0000","last_active":"2024-03-10T03:50:08.686273+0000","last_peered":"2024-03-10T03:50:08.686273+0000","last_clean":"2024-03-10T03:50:08.686273+0000","last_became_active":"2024-03-10T03:49:29.225430+0000","last_became_peered":"2024-03-10T03:49:29.225430+0000","last_unstale":"2024-03-10T03:50:08.686273+0000","last_undegraded":"2024-03-10T03:50:08.686273+0000","last_fullsized":"2024-03-10T03:50:08.686273+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:18:44.898443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"69'1","reported_seq":18,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676976+0000","last_change":"2024-03-10T03:49:31.239529+0000","last_active":"2024-03-10T03:50:08.676976+0000","last_peered":"2024-03-10T03:50:08.676976+0000","last_clean":"2024-03-10T03:50:08.676976+0000","last_became_active":"2024-03-10T03:49:31.239196+0000","last_became_peered":"2024-03-10T03:49:31.239196+0000","last_unstale":"2024-03-10T03:50:08.676976+0000","last_undegraded":"2024-03-10T03:50:08.676976+0000","last_fullsized":"2024-03-10T03:50:08.676976+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:21:35.680393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"71'188","reported_seq":344,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:10.734018+0000","last_change":"2024-03-10T03:49:12.961905+0000","last_active":"2024-03-10T03:50:10.734018+0000","last_peered":"2024-03-10T03:50:10.734018+0000","last_clean":"2024-03-10T03:50:10.734018+0000","last_became_active":"2024-03-10T03:49:12.961515+0000","last_became_peered":"2024-03-10T03:49:12.961515+0000","last_unstale":"2024-03-10T03:50:10.734018+0000","last_undegraded":"2024-03-10T03:50:10.734018+0000","last_fullsized":"2024-03-10T03:50:10.734018+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:47:01.843240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:47:01.843240+0000","last_clean_scrub_stamp":"2024-03-10T03:47:01.843240+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:25:33.117670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767730+0000","last_change":"2024-03-10T03:49:25.210934+0000","last_active":"2024-03-10T03:50:08.767730+0000","last_peered":"2024-03-10T03:50:08.767730+0000","last_clean":"2024-03-10T03:50:08.767730+0000","last_became_active":"2024-03-10T03:49:25.208887+0000","last_became_peered":"2024-03-10T03:49:25.208887+0000","last_unstale":"2024-03-10T03:50:08.767730+0000","last_undegraded":"2024-03-10T03:50:08.767730+0000","last_fullsized":"2024-03-10T03:50:08.767730+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:22:37.687467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"69'16","reported_seq":66,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:50.272930+0000","last_change":"2024-03-10T03:49:27.224799+0000","last_active":"2024-03-10T03:50:50.272930+0000","last_peered":"2024-03-10T03:50:50.272930+0000","last_clean":"2024-03-10T03:50:50.272930+0000","last_became_active":"2024-03-10T03:49:27.224486+0000","last_became_peered":"2024-03-10T03:49:27.224486+0000","last_unstale":"2024-03-10T03:50:50.272930+0000","last_undegraded":"2024-03-10T03:50:50.272930+0000","last_fullsized":"2024-03-10T03:50:50.272930+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:12:02.696623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558708+0000","last_change":"2024-03-10T03:49:29.231568+0000","last_active":"2024-03-10T03:50:08.558708+0000","last_peered":"2024-03-10T03:50:08.558708+0000","last_clean":"2024-03-10T03:50:08.558708+0000","last_became_active":"2024-03-10T03:49:29.231267+0000","last_became_peered":"2024-03-10T03:49:29.231267+0000","last_unstale":"2024-03-10T03:50:08.558708+0000","last_undegraded":"2024-03-10T03:50:08.558708+0000","last_fullsized":"2024-03-10T03:50:08.558708+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:26:21.829789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.559745+0000","last_change":"2024-03-10T03:49:31.240018+0000","last_active":"2024-03-10T03:50:08.559745+0000","last_peered":"2024-03-10T03:50:08.559745+0000","last_clean":"2024-03-10T03:50:08.559745+0000","last_became_active":"2024-03-10T03:49:31.239699+0000","last_became_peered":"2024-03-10T03:49:31.239699+0000","last_unstale":"2024-03-10T03:50:08.559745+0000","last_undegraded":"2024-03-10T03:50:08.559745+0000","last_fullsized":"2024-03-10T03:50:08.559745+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:58:01.177798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"62'1","reported_seq":37,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.556303+0000","last_change":"2024-03-10T03:49:25.217129+0000","last_active":"2024-03-10T03:50:08.556303+0000","last_peered":"2024-03-10T03:50:08.556303+0000","last_clean":"2024-03-10T03:50:08.556303+0000","last_became_active":"2024-03-10T03:49:25.216788+0000","last_became_peered":"2024-03-10T03:49:25.216788+0000","last_unstale":"2024-03-10T03:50:08.556303+0000","last_undegraded":"2024-03-10T03:50:08.556303+0000","last_fullsized":"2024-03-10T03:50:08.556303+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:08:45.529042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"70'5","reported_seq":125,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:57.914266+0000","last_change":"2024-03-10T03:49:29.223404+0000","last_active":"2024-03-10T03:50:57.914266+0000","last_peered":"2024-03-10T03:50:57.914266+0000","last_clean":"2024-03-10T03:50:57.914266+0000","last_became_active":"2024-03-10T03:49:25.210697+0000","last_became_peered":"2024-03-10T03:49:25.210697+0000","last_unstale":"2024-03-10T03:50:57.914266+0000","last_undegraded":"2024-03-10T03:50:57.914266+0000","last_fullsized":"2024-03-10T03:50:57.914266+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:03:44.881734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011382409999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":86,"num_read_kb":81,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"69'12","reported_seq":43,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.556766+0000","last_change":"2024-03-10T03:49:27.214011+0000","last_active":"2024-03-10T03:50:08.556766+0000","last_peered":"2024-03-10T03:50:08.556766+0000","last_clean":"2024-03-10T03:50:08.556766+0000","last_became_active":"2024-03-10T03:49:27.213719+0000","last_became_peered":"2024-03-10T03:49:27.213719+0000","last_unstale":"2024-03-10T03:50:08.556766+0000","last_undegraded":"2024-03-10T03:50:08.556766+0000","last_fullsized":"2024-03-10T03:50:08.556766+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:13:56.642032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.677131+0000","last_change":"2024-03-10T03:49:29.228840+0000","last_active":"2024-03-10T03:50:08.677131+0000","last_peered":"2024-03-10T03:50:08.677131+0000","last_clean":"2024-03-10T03:50:08.677131+0000","last_became_active":"2024-03-10T03:49:29.228546+0000","last_became_peered":"2024-03-10T03:49:29.228546+0000","last_unstale":"2024-03-10T03:50:08.677131+0000","last_undegraded":"2024-03-10T03:50:08.677131+0000","last_fullsized":"2024-03-10T03:50:08.677131+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:11:22.484853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557920+0000","last_change":"2024-03-10T03:49:31.303021+0000","last_active":"2024-03-10T03:50:08.557920+0000","last_peered":"2024-03-10T03:50:08.557920+0000","last_clean":"2024-03-10T03:50:08.557920+0000","last_became_active":"2024-03-10T03:49:31.302599+0000","last_became_peered":"2024-03-10T03:49:31.302599+0000","last_unstale":"2024-03-10T03:50:08.557920+0000","last_undegraded":"2024-03-10T03:50:08.557920+0000","last_fullsized":"2024-03-10T03:50:08.557920+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:35:25.934917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.559996+0000","last_change":"2024-03-10T03:49:25.201942+0000","last_active":"2024-03-10T03:50:08.559996+0000","last_peered":"2024-03-10T03:50:08.559996+0000","last_clean":"2024-03-10T03:50:08.559996+0000","last_became_active":"2024-03-10T03:49:25.201605+0000","last_became_peered":"2024-03-10T03:49:25.201605+0000","last_unstale":"2024-03-10T03:50:08.559996+0000","last_undegraded":"2024-03-10T03:50:08.559996+0000","last_fullsized":"2024-03-10T03:50:08.559996+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:06:06.152147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"69'25","reported_seq":73,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.557220+0000","last_change":"2024-03-10T03:49:27.225667+0000","last_active":"2024-03-10T03:50:08.557220+0000","last_peered":"2024-03-10T03:50:08.557220+0000","last_clean":"2024-03-10T03:50:08.557220+0000","last_became_active":"2024-03-10T03:49:27.225224+0000","last_became_peered":"2024-03-10T03:49:27.225224+0000","last_unstale":"2024-03-10T03:50:08.557220+0000","last_undegraded":"2024-03-10T03:50:08.557220+0000","last_fullsized":"2024-03-10T03:50:08.557220+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:03:00.490166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767642+0000","last_change":"2024-03-10T03:49:29.225780+0000","last_active":"2024-03-10T03:50:08.767642+0000","last_peered":"2024-03-10T03:50:08.767642+0000","last_clean":"2024-03-10T03:50:08.767642+0000","last_became_active":"2024-03-10T03:49:29.225488+0000","last_became_peered":"2024-03-10T03:49:29.225488+0000","last_unstale":"2024-03-10T03:50:08.767642+0000","last_undegraded":"2024-03-10T03:50:08.767642+0000","last_fullsized":"2024-03-10T03:50:08.767642+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:54:58.916740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.555911+0000","last_change":"2024-03-10T03:49:31.243419+0000","last_active":"2024-03-10T03:50:08.555911+0000","last_peered":"2024-03-10T03:50:08.555911+0000","last_clean":"2024-03-10T03:50:08.555911+0000","last_became_active":"2024-03-10T03:49:31.243070+0000","last_became_peered":"2024-03-10T03:49:31.243070+0000","last_unstale":"2024-03-10T03:50:08.555911+0000","last_undegraded":"2024-03-10T03:50:08.555911+0000","last_fullsized":"2024-03-10T03:50:08.555911+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T10:03:13.208872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676802+0000","last_change":"2024-03-10T03:49:25.211684+0000","last_active":"2024-03-10T03:50:08.676802+0000","last_peered":"2024-03-10T03:50:08.676802+0000","last_clean":"2024-03-10T03:50:08.676802+0000","last_became_active":"2024-03-10T03:49:25.210379+0000","last_became_peered":"2024-03-10T03:49:25.210379+0000","last_unstale":"2024-03-10T03:50:08.676802+0000","last_undegraded":"2024-03-10T03:50:08.676802+0000","last_fullsized":"2024-03-10T03:50:08.676802+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:46:09.417306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"69'16","reported_seq":52,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.559705+0000","last_change":"2024-03-10T03:49:27.211176+0000","last_active":"2024-03-10T03:50:08.559705+0000","last_peered":"2024-03-10T03:50:08.559705+0000","last_clean":"2024-03-10T03:50:08.559705+0000","last_became_active":"2024-03-10T03:49:27.210864+0000","last_became_peered":"2024-03-10T03:49:27.210864+0000","last_unstale":"2024-03-10T03:50:08.559705+0000","last_undegraded":"2024-03-10T03:50:08.559705+0000","last_fullsized":"2024-03-10T03:50:08.559705+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:35:32.001856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"70'11","reported_seq":51,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:50.272673+0000","last_change":"2024-03-10T03:49:29.226373+0000","last_active":"2024-03-10T03:50:50.272673+0000","last_peered":"2024-03-10T03:50:50.272673+0000","last_clean":"2024-03-10T03:50:50.272673+0000","last_became_active":"2024-03-10T03:49:29.226090+0000","last_became_peered":"2024-03-10T03:49:29.226090+0000","last_unstale":"2024-03-10T03:50:50.272673+0000","last_undegraded":"2024-03-10T03:50:50.272673+0000","last_fullsized":"2024-03-10T03:50:50.272673+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:59:07.060353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558344+0000","last_change":"2024-03-10T03:49:31.304476+0000","last_active":"2024-03-10T03:50:08.558344+0000","last_peered":"2024-03-10T03:50:08.558344+0000","last_clean":"2024-03-10T03:50:08.558344+0000","last_became_active":"2024-03-10T03:49:31.304170+0000","last_became_peered":"2024-03-10T03:49:31.304170+0000","last_unstale":"2024-03-10T03:50:08.558344+0000","last_undegraded":"2024-03-10T03:50:08.558344+0000","last_fullsized":"2024-03-10T03:50:08.558344+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:22:35.639526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"62'1","reported_seq":30,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.556893+0000","last_change":"2024-03-10T03:49:25.214344+0000","last_active":"2024-03-10T03:50:08.556893+0000","last_peered":"2024-03-10T03:50:08.556893+0000","last_clean":"2024-03-10T03:50:08.556893+0000","last_became_active":"2024-03-10T03:49:25.214034+0000","last_became_peered":"2024-03-10T03:49:25.214034+0000","last_unstale":"2024-03-10T03:50:08.556893+0000","last_undegraded":"2024-03-10T03:50:08.556893+0000","last_fullsized":"2024-03-10T03:50:08.556893+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T15:18:20.722043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"69'20","reported_seq":58,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767972+0000","last_change":"2024-03-10T03:49:27.211589+0000","last_active":"2024-03-10T03:50:08.767972+0000","last_peered":"2024-03-10T03:50:08.767972+0000","last_clean":"2024-03-10T03:50:08.767972+0000","last_became_active":"2024-03-10T03:49:27.211059+0000","last_became_peered":"2024-03-10T03:49:27.211059+0000","last_unstale":"2024-03-10T03:50:08.767972+0000","last_undegraded":"2024-03-10T03:50:08.767972+0000","last_fullsized":"2024-03-10T03:50:08.767972+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:37:14.305807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"70'11","reported_seq":51,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:50.272684+0000","last_change":"2024-03-10T03:49:29.221225+0000","last_active":"2024-03-10T03:50:50.272684+0000","last_peered":"2024-03-10T03:50:50.272684+0000","last_clean":"2024-03-10T03:50:50.272684+0000","last_became_active":"2024-03-10T03:49:29.220806+0000","last_became_peered":"2024-03-10T03:49:29.220806+0000","last_unstale":"2024-03-10T03:50:50.272684+0000","last_undegraded":"2024-03-10T03:50:50.272684+0000","last_fullsized":"2024-03-10T03:50:50.272684+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T12:36:51.097675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768123+0000","last_change":"2024-03-10T03:49:31.238598+0000","last_active":"2024-03-10T03:50:08.768123+0000","last_peered":"2024-03-10T03:50:08.768123+0000","last_clean":"2024-03-10T03:50:08.768123+0000","last_became_active":"2024-03-10T03:49:31.238307+0000","last_became_peered":"2024-03-10T03:49:31.238307+0000","last_unstale":"2024-03-10T03:50:08.768123+0000","last_undegraded":"2024-03-10T03:50:08.768123+0000","last_fullsized":"2024-03-10T03:50:08.768123+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:31:59.005442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768599+0000","last_change":"2024-03-10T03:49:25.217775+0000","last_active":"2024-03-10T03:50:08.768599+0000","last_peered":"2024-03-10T03:50:08.768599+0000","last_clean":"2024-03-10T03:50:08.768599+0000","last_became_active":"2024-03-10T03:49:25.217424+0000","last_became_peered":"2024-03-10T03:49:25.217424+0000","last_unstale":"2024-03-10T03:50:08.768599+0000","last_undegraded":"2024-03-10T03:50:08.768599+0000","last_fullsized":"2024-03-10T03:50:08.768599+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T11:28:15.980479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"69'30","reported_seq":81,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767456+0000","last_change":"2024-03-10T03:49:27.225474+0000","last_active":"2024-03-10T03:50:08.767456+0000","last_peered":"2024-03-10T03:50:08.767456+0000","last_clean":"2024-03-10T03:50:08.767456+0000","last_became_active":"2024-03-10T03:49:27.225207+0000","last_became_peered":"2024-03-10T03:49:27.225207+0000","last_unstale":"2024-03-10T03:50:08.767456+0000","last_undegraded":"2024-03-10T03:50:08.767456+0000","last_fullsized":"2024-03-10T03:50:08.767456+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:35:15.031216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767783+0000","last_change":"2024-03-10T03:49:29.223283+0000","last_active":"2024-03-10T03:50:08.767783+0000","last_peered":"2024-03-10T03:50:08.767783+0000","last_clean":"2024-03-10T03:50:08.767783+0000","last_became_active":"2024-03-10T03:49:29.222977+0000","last_became_peered":"2024-03-10T03:49:29.222977+0000","last_unstale":"2024-03-10T03:50:08.767783+0000","last_undegraded":"2024-03-10T03:50:08.767783+0000","last_fullsized":"2024-03-10T03:50:08.767783+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T15:08:12.040216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.676676+0000","last_change":"2024-03-10T03:49:31.231766+0000","last_active":"2024-03-10T03:50:08.676676+0000","last_peered":"2024-03-10T03:50:08.676676+0000","last_clean":"2024-03-10T03:50:08.676676+0000","last_became_active":"2024-03-10T03:49:31.231486+0000","last_became_peered":"2024-03-10T03:49:31.231486+0000","last_unstale":"2024-03-10T03:50:08.676676+0000","last_undegraded":"2024-03-10T03:50:08.676676+0000","last_fullsized":"2024-03-10T03:50:08.676676+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T14:09:55.084176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"70'11","reported_seq":51,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:50.272688+0000","last_change":"2024-03-10T03:49:29.224677+0000","last_active":"2024-03-10T03:50:50.272688+0000","last_peered":"2024-03-10T03:50:50.272688+0000","last_clean":"2024-03-10T03:50:50.272688+0000","last_became_active":"2024-03-10T03:49:29.224368+0000","last_became_peered":"2024-03-10T03:49:29.224368+0000","last_unstale":"2024-03-10T03:50:50.272688+0000","last_undegraded":"2024-03-10T03:50:50.272688+0000","last_fullsized":"2024-03-10T03:50:50.272688+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:32:13.390081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767376+0000","last_change":"2024-03-10T03:49:25.213828+0000","last_active":"2024-03-10T03:50:08.767376+0000","last_peered":"2024-03-10T03:50:08.767376+0000","last_clean":"2024-03-10T03:50:08.767376+0000","last_became_active":"2024-03-10T03:49:25.213523+0000","last_became_peered":"2024-03-10T03:49:25.213523+0000","last_unstale":"2024-03-10T03:50:08.767376+0000","last_undegraded":"2024-03-10T03:50:08.767376+0000","last_fullsized":"2024-03-10T03:50:08.767376+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:04:01.802567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"69'15","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.558700+0000","last_change":"2024-03-10T03:49:27.215761+0000","last_active":"2024-03-10T03:50:08.558700+0000","last_peered":"2024-03-10T03:50:08.558700+0000","last_clean":"2024-03-10T03:50:08.558700+0000","last_became_active":"2024-03-10T03:49:27.215390+0000","last_became_peered":"2024-03-10T03:49:27.215390+0000","last_unstale":"2024-03-10T03:50:08.558700+0000","last_undegraded":"2024-03-10T03:50:08.558700+0000","last_fullsized":"2024-03-10T03:50:08.558700+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T04:36:27.724770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767910+0000","last_change":"2024-03-10T03:49:31.303033+0000","last_active":"2024-03-10T03:50:08.767910+0000","last_peered":"2024-03-10T03:50:08.767910+0000","last_clean":"2024-03-10T03:50:08.767910+0000","last_became_active":"2024-03-10T03:49:31.302665+0000","last_became_peered":"2024-03-10T03:49:31.302665+0000","last_unstale":"2024-03-10T03:50:08.767910+0000","last_undegraded":"2024-03-10T03:50:08.767910+0000","last_fullsized":"2024-03-10T03:50:08.767910+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:52:53.372407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.677125+0000","last_change":"2024-03-10T03:49:29.231051+0000","last_active":"2024-03-10T03:50:08.677125+0000","last_peered":"2024-03-10T03:50:08.677125+0000","last_clean":"2024-03-10T03:50:08.677125+0000","last_became_active":"2024-03-10T03:49:29.230777+0000","last_became_peered":"2024-03-10T03:49:29.230777+0000","last_unstale":"2024-03-10T03:50:08.677125+0000","last_undegraded":"2024-03-10T03:50:08.677125+0000","last_fullsized":"2024-03-10T03:50:08.677125+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T09:19:30.050107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767846+0000","last_change":"2024-03-10T03:49:25.210285+0000","last_active":"2024-03-10T03:50:08.767846+0000","last_peered":"2024-03-10T03:50:08.767846+0000","last_clean":"2024-03-10T03:50:08.767846+0000","last_became_active":"2024-03-10T03:49:25.209986+0000","last_became_peered":"2024-03-10T03:49:25.209986+0000","last_unstale":"2024-03-10T03:50:08.767846+0000","last_undegraded":"2024-03-10T03:50:08.767846+0000","last_fullsized":"2024-03-10T03:50:08.767846+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T06:05:13.072307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"69'17","reported_seq":56,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768750+0000","last_change":"2024-03-10T03:49:27.209375+0000","last_active":"2024-03-10T03:50:08.768750+0000","last_peered":"2024-03-10T03:50:08.768750+0000","last_clean":"2024-03-10T03:50:08.768750+0000","last_became_active":"2024-03-10T03:49:27.209075+0000","last_became_peered":"2024-03-10T03:49:27.209075+0000","last_unstale":"2024-03-10T03:50:08.768750+0000","last_undegraded":"2024-03-10T03:50:08.768750+0000","last_fullsized":"2024-03-10T03:50:08.768750+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:17:50.344799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.686250+0000","last_change":"2024-03-10T03:49:31.240710+0000","last_active":"2024-03-10T03:50:08.686250+0000","last_peered":"2024-03-10T03:50:08.686250+0000","last_clean":"2024-03-10T03:50:08.686250+0000","last_became_active":"2024-03-10T03:49:31.240378+0000","last_became_peered":"2024-03-10T03:49:31.240378+0000","last_unstale":"2024-03-10T03:50:08.686250+0000","last_undegraded":"2024-03-10T03:50:08.686250+0000","last_fullsized":"2024-03-10T03:50:08.686250+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:52:39.392944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.560163+0000","last_change":"2024-03-10T03:49:29.225948+0000","last_active":"2024-03-10T03:50:08.560163+0000","last_peered":"2024-03-10T03:50:08.560163+0000","last_clean":"2024-03-10T03:50:08.560163+0000","last_became_active":"2024-03-10T03:49:29.225621+0000","last_became_peered":"2024-03-10T03:49:29.225621+0000","last_unstale":"2024-03-10T03:50:08.560163+0000","last_undegraded":"2024-03-10T03:50:08.560163+0000","last_fullsized":"2024-03-10T03:50:08.560163+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:54:34.072463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.767743+0000","last_change":"2024-03-10T03:49:25.207108+0000","last_active":"2024-03-10T03:50:08.767743+0000","last_peered":"2024-03-10T03:50:08.767743+0000","last_clean":"2024-03-10T03:50:08.767743+0000","last_became_active":"2024-03-10T03:49:25.206813+0000","last_became_peered":"2024-03-10T03:49:25.206813+0000","last_unstale":"2024-03-10T03:50:08.767743+0000","last_undegraded":"2024-03-10T03:50:08.767743+0000","last_fullsized":"2024-03-10T03:50:08.767743+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:18:59.642192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"69'17","reported_seq":53,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768686+0000","last_change":"2024-03-10T03:49:27.207609+0000","last_active":"2024-03-10T03:50:08.768686+0000","last_peered":"2024-03-10T03:50:08.768686+0000","last_clean":"2024-03-10T03:50:08.768686+0000","last_became_active":"2024-03-10T03:49:27.207265+0000","last_became_peered":"2024-03-10T03:49:27.207265+0000","last_unstale":"2024-03-10T03:50:08.768686+0000","last_undegraded":"2024-03-10T03:50:08.768686+0000","last_fullsized":"2024-03-10T03:50:08.768686+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T05:19:45.912429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":17,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768892+0000","last_change":"2024-03-10T03:49:31.238754+0000","last_active":"2024-03-10T03:50:08.768892+0000","last_peered":"2024-03-10T03:50:08.768892+0000","last_clean":"2024-03-10T03:50:08.768892+0000","last_became_active":"2024-03-10T03:49:31.238441+0000","last_became_peered":"2024-03-10T03:49:31.238441+0000","last_unstale":"2024-03-10T03:50:08.768892+0000","last_undegraded":"2024-03-10T03:50:08.768892+0000","last_fullsized":"2024-03-10T03:50:08.768892+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:30.207653+0000","last_clean_scrub_stamp":"2024-03-10T03:49:30.207653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T13:54:31.344185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"70'11","reported_seq":51,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:50.272673+0000","last_change":"2024-03-10T03:49:29.229294+0000","last_active":"2024-03-10T03:50:50.272673+0000","last_peered":"2024-03-10T03:50:50.272673+0000","last_clean":"2024-03-10T03:50:50.272673+0000","last_became_active":"2024-03-10T03:49:29.228967+0000","last_became_peered":"2024-03-10T03:49:29.228967+0000","last_unstale":"2024-03-10T03:50:50.272673+0000","last_undegraded":"2024-03-10T03:50:50.272673+0000","last_fullsized":"2024-03-10T03:50:50.272673+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:28.199287+0000","last_clean_scrub_stamp":"2024-03-10T03:49:28.199287+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T08:49:59.707748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768053+0000","last_change":"2024-03-10T03:49:25.218892+0000","last_active":"2024-03-10T03:50:08.768053+0000","last_peered":"2024-03-10T03:50:08.768053+0000","last_clean":"2024-03-10T03:50:08.768053+0000","last_became_active":"2024-03-10T03:49:25.218608+0000","last_became_peered":"2024-03-10T03:49:25.218608+0000","last_unstale":"2024-03-10T03:50:08.768053+0000","last_undegraded":"2024-03-10T03:50:08.768053+0000","last_fullsized":"2024-03-10T03:50:08.768053+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:24.186207+0000","last_clean_scrub_stamp":"2024-03-10T03:49:24.186207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T15:04:14.178696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"69'10","reported_seq":40,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-10T03:50:08.768290+0000","last_change":"2024-03-10T03:49:27.217125+0000","last_active":"2024-03-10T03:50:08.768290+0000","last_peered":"2024-03-10T03:50:08.768290+0000","last_clean":"2024-03-10T03:50:08.768290+0000","last_became_active":"2024-03-10T03:49:27.216833+0000","last_became_peered":"2024-03-10T03:49:27.216833+0000","last_unstale":"2024-03-10T03:50:08.768290+0000","last_undegraded":"2024-03-10T03:50:08.768290+0000","last_fullsized":"2024-03-10T03:50:08.768290+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-10T03:49:26.196238+0000","last_clean_scrub_stamp":"2024-03-10T03:49:26.196238+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T07:35:16.280725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":989,"num_read_kb":714,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":589,"ondisk_log_size":589,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":333,"num_write":245,"num_write_kb":3982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":188,"ondisk_log_size":188,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":81,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168601,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":28372,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93712684,"statfs":{"total":95990841344,"available":95961788416,"internally_reserved":0,"allocated":1388544,"data_stored":916599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430237,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32512,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708544,"statfs":{"total":95990841344,"available":95957549056,"internally_reserved":0,"allocated":1368064,"data_stored":915547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31996,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709060,"statfs":{"total":95990841344,"available":95958077440,"internally_reserved":0,"allocated":835584,"data_stored":326089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738430,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":28376,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93712680,"statfs":{"total":95990841344,"available":95961784320,"internally_reserved":0,"allocated":1392640,"data_stored":915864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411379,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31928,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709128,"statfs":{"total":95990841344,"available":95958147072,"internally_reserved":0,"allocated":765952,"data_stored":324419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149741,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93713240,"statfs":{"total":95990841344,"available":95962357760,"internally_reserved":0,"allocated":815104,"data_stored":325606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986216,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31972,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709084,"statfs":{"total":95990841344,"available":95958102016,"internally_reserved":0,"allocated":811008,"data_stored":326195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724579,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":31940,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93709116,"statfs":{"total":95990841344,"available":95958134784,"internally_reserved":0,"allocated":778240,"data_stored":324643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-10T03:51:04.807 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-10T03:51:04.807 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-10T03:51:04.807 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-10T03:51:04.807 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- ceph health --format=json 2024-03-10T03:51:05.048 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:04 smithi006 podman[61556]: 2024-03-10 03:51:04.925143964 +0000 UTC m=+4.237310391 container create 35152e971ef48bfd9af588dca34d6e25af3f38185241106ca353b6536b1a6f56 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-10T03:51:05.343 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:51:05.474 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 podman[61556]: 2024-03-10 03:51:05.169117899 +0000 UTC m=+4.481284325 container init 35152e971ef48bfd9af588dca34d6e25af3f38185241106ca353b6536b1a6f56 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-10T03:51:05.474 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 podman[61556]: 2024-03-10 03:51:05.171838349 +0000 UTC m=+4.484004784 container start 35152e971ef48bfd9af588dca34d6e25af3f38185241106ca353b6536b1a6f56 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-10T03:51:05.474 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.175Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-10T03:51:05.474 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.175Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-10T03:51:05.474 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.176Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-10T03:51:05.474 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.176Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-10T03:51:05.474 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.176Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-10T03:51:05.475 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.176Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-10T03:51:05.475 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.176Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-10T03:51:05.475 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.176Z caller=node_exporter.go:117 level=info collector=arp 2024-03-10T03:51:05.475 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.176Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-10T03:51:05.475 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.176Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-10T03:51:05.475 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-10T03:51:05.475 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-10T03:51:05.475 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-10T03:51:05.475 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-10T03:51:05.475 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-10T03:51:05.475 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-10T03:51:05.475 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=edac 2024-03-10T03:51:05.475 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-10T03:51:05.475 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-10T03:51:05.476 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-10T03:51:05.476 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-10T03:51:05.476 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-10T03:51:05.476 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-10T03:51:05.476 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-10T03:51:05.476 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-10T03:51:05.476 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-10T03:51:05.476 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-10T03:51:05.476 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-10T03:51:05.476 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-10T03:51:05.476 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-10T03:51:05.476 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-10T03:51:05.476 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-10T03:51:05.477 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-10T03:51:05.477 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=os 2024-03-10T03:51:05.477 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-10T03:51:05.477 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-10T03:51:05.477 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-10T03:51:05.477 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-10T03:51:05.477 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-10T03:51:05.477 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-10T03:51:05.477 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-10T03:51:05.477 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=stat 2024-03-10T03:51:05.477 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-10T03:51:05.477 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-10T03:51:05.477 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-10T03:51:05.477 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=time 2024-03-10T03:51:05.477 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-10T03:51:05.478 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=uname 2024-03-10T03:51:05.478 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-10T03:51:05.478 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-10T03:51:05.478 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-10T03:51:05.478 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-10T03:51:05.478 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a[61863]: ts=2024-03-10T03:51:05.177Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-10T03:51:05.478 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 bash[61556]: 35152e971ef48bfd9af588dca34d6e25af3f38185241106ca353b6536b1a6f56 2024-03-10T03:51:05.478 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:51:05 smithi006 systemd[1]: Started Ceph node-exporter.a for 414ba1e0-de90-11ee-95c7-87774f69a715. 2024-03-10T03:51:06.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:06 smithi006 ceph-mon[25256]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:51:06.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:06 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:06.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:06 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:06.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:06 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:06.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:06 smithi006 ceph-mon[25256]: Deploying daemon node-exporter.b on smithi071 2024-03-10T03:51:06.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:06 smithi006 ceph-mon[31842]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:51:06.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:06 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:06.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:06 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:06.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:06 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:06.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:06 smithi006 ceph-mon[31842]: Deploying daemon node-exporter.b on smithi071 2024-03-10T03:51:06.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:06 smithi071 ceph-mon[27181]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:51:06.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:06 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:06.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:06 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:06.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:06 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:06.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:06 smithi071 ceph-mon[27181]: Deploying daemon node-exporter.b on smithi071 2024-03-10T03:51:06.591 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:06 smithi071 bash[52199]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-10T03:51:07.340 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:06 smithi071 bash[52199]: Getting image source signatures 2024-03-10T03:51:07.341 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:06 smithi071 bash[52199]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-10T03:51:07.341 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:06 smithi071 bash[52199]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-10T03:51:07.341 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:06 smithi071 bash[52199]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-10T03:51:08.144 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-10T03:51:08.144 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-10T03:51:08.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:08 smithi006 ceph-mon[25256]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:51:08.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:08 smithi006 ceph-mon[25256]: from='client.? 172.21.15.6:0/1799952296' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-10T03:51:08.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:08 smithi006 ceph-mon[31842]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:51:08.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:08 smithi006 ceph-mon[31842]: from='client.? 172.21.15.6:0/1799952296' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-10T03:51:08.590 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:08 smithi071 ceph-mon[27181]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:51:08.590 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:08 smithi071 ceph-mon[27181]: from='client.? 172.21.15.6:0/1799952296' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-10T03:51:08.590 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:08 smithi071 bash[52199]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-10T03:51:08.591 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:08 smithi071 bash[52199]: Writing manifest to image destination 2024-03-10T03:51:08.984 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-10T03:51:08.984 INFO:tasks.cephadm:Setup complete, yielding 2024-03-10T03:51:08.984 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-10T03:51:08.994 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi006.front.sepia.ceph.com 2024-03-10T03:51:08.994 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-03-10T03:51:08.997 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:08 smithi071 podman[52199]: 2024-03-10 03:51:08.646883267 +0000 UTC m=+2.211872061 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-10T03:51:08.997 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:08 smithi071 podman[52199]: 2024-03-10 03:51:08.745446799 +0000 UTC m=+2.310435624 container create b8040739857f947aa2ca5a899c8722897eb3053274d92a18a9dcff5f68090cbe (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-10T03:51:09.323 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:08 smithi071 podman[52199]: 2024-03-10 03:51:08.997548532 +0000 UTC m=+2.562537326 container init b8040739857f947aa2ca5a899c8722897eb3053274d92a18a9dcff5f68090cbe (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-10T03:51:09.323 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 podman[52199]: 2024-03-10 03:51:09.000512475 +0000 UTC m=+2.565501270 container start b8040739857f947aa2ca5a899c8722897eb3053274d92a18a9dcff5f68090cbe (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-10T03:51:09.323 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.005Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-10T03:51:09.323 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.005Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-10T03:51:09.323 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.005Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-10T03:51:09.323 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.005Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-10T03:51:09.323 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-10T03:51:09.323 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-10T03:51:09.323 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-10T03:51:09.323 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:117 level=info collector=arp 2024-03-10T03:51:09.323 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-10T03:51:09.324 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-10T03:51:09.324 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-10T03:51:09.324 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-10T03:51:09.324 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-10T03:51:09.324 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-10T03:51:09.324 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-10T03:51:09.324 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-10T03:51:09.324 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:117 level=info collector=edac 2024-03-10T03:51:09.325 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-10T03:51:09.325 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-10T03:51:09.325 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-10T03:51:09.325 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-10T03:51:09.325 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-10T03:51:09.325 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-10T03:51:09.325 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-10T03:51:09.325 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-10T03:51:09.325 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-10T03:51:09.325 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-10T03:51:09.325 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-10T03:51:09.325 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-10T03:51:09.325 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-10T03:51:09.325 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.006Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-10T03:51:09.326 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.007Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-10T03:51:09.326 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.007Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-10T03:51:09.326 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.007Z caller=node_exporter.go:117 level=info collector=os 2024-03-10T03:51:09.326 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.007Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-10T03:51:09.326 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.007Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-10T03:51:09.326 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.007Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-10T03:51:09.326 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.007Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-10T03:51:09.326 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.007Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-10T03:51:09.326 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.007Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-10T03:51:09.326 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.007Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-10T03:51:09.327 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.007Z caller=node_exporter.go:117 level=info collector=stat 2024-03-10T03:51:09.327 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.007Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-10T03:51:09.327 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.007Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-10T03:51:09.327 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.007Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-10T03:51:09.327 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.007Z caller=node_exporter.go:117 level=info collector=time 2024-03-10T03:51:09.327 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.007Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-10T03:51:09.327 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.007Z caller=node_exporter.go:117 level=info collector=uname 2024-03-10T03:51:09.327 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.007Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-10T03:51:09.327 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.007Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-10T03:51:09.327 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.007Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-10T03:51:09.327 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.007Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-10T03:51:09.327 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b[52249]: ts=2024-03-10T03:51:09.007Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-10T03:51:09.327 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 bash[52199]: b8040739857f947aa2ca5a899c8722897eb3053274d92a18a9dcff5f68090cbe 2024-03-10T03:51:09.327 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:51:09 smithi071 systemd[1]: Started Ceph node-exporter.b for 414ba1e0-de90-11ee-95c7-87774f69a715. 2024-03-10T03:51:09.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 5]}]: dispatch 2024-03-10T03:51:09.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-10T03:51:09.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:09 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 5]}]: dispatch 2024-03-10T03:51:09.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-10T03:51:09.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:09 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-10T03:51:09.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:09 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-10T03:51:09.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:09 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T03:51:09.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:09 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:09.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:09 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:09.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:09 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:09.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:09 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:09.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:09 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:09.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 5]}]: dispatch 2024-03-10T03:51:09.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-10T03:51:09.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:09 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 5]}]: dispatch 2024-03-10T03:51:09.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-10T03:51:09.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:09 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-10T03:51:09.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:09 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-10T03:51:09.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:09 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T03:51:09.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:09 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:09.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:09 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:09.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:09 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:09.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:09 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:09.588 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:09 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:09.590 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 5]}]: dispatch 2024-03-10T03:51:09.590 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-10T03:51:09.590 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 5]}]: dispatch 2024-03-10T03:51:09.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-10T03:51:09.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-10T03:51:09.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-10T03:51:09.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T03:51:09.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:09.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:09.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:09.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:09.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:09.820 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:51:09.901 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:51:09 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a[47087]: debug there is no tcmu-runner data available 2024-03-10T03:51:10.531 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:10 smithi071 ceph-mon[27181]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:51:10.531 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:10 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 5]}]': finished 2024-03-10T03:51:10.531 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:10 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-10T03:51:10.531 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:10 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-03-10T03:51:10.531 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:10 smithi071 ceph-mon[27181]: osdmap e72: 8 total, 8 up, 8 in 2024-03-10T03:51:10.531 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:10 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:51:10.531 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:10 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi071", "caps": []}]: dispatch 2024-03-10T03:51:10.531 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:10 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi006", "caps": []}]: dispatch 2024-03-10T03:51:10.531 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:10 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi071", "caps": []}]: dispatch 2024-03-10T03:51:10.531 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:10 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi006", "caps": []}]: dispatch 2024-03-10T03:51:10.561 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:10 smithi006 ceph-mon[25256]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:51:10.561 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:10 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 5]}]': finished 2024-03-10T03:51:10.561 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:10 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-10T03:51:10.561 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:10 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-03-10T03:51:10.561 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:10 smithi006 ceph-mon[25256]: osdmap e72: 8 total, 8 up, 8 in 2024-03-10T03:51:10.561 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:10 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:51:10.561 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:10 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi071", "caps": []}]: dispatch 2024-03-10T03:51:10.561 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:10 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi006", "caps": []}]: dispatch 2024-03-10T03:51:10.561 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:10 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi071", "caps": []}]: dispatch 2024-03-10T03:51:10.561 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:10 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi006", "caps": []}]: dispatch 2024-03-10T03:51:10.562 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:10 smithi006 ceph-mon[31842]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:51:10.562 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:10 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 5]}]': finished 2024-03-10T03:51:10.562 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:10 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-10T03:51:10.562 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:10 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-03-10T03:51:10.562 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:10 smithi006 ceph-mon[31842]: osdmap e72: 8 total, 8 up, 8 in 2024-03-10T03:51:10.562 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:10 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:51:10.562 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:10 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi071", "caps": []}]: dispatch 2024-03-10T03:51:10.562 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:10 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi006", "caps": []}]: dispatch 2024-03-10T03:51:10.562 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:10 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi071", "caps": []}]: dispatch 2024-03-10T03:51:10.562 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:10 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi006", "caps": []}]: dispatch 2024-03-10T03:51:11.541 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:11 smithi006 ceph-mon[25256]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:51:11.542 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:11 smithi006 ceph-mon[25256]: Reconfiguring daemon agent.smithi006 on smithi006 2024-03-10T03:51:11.542 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:11 smithi006 ceph-mon[25256]: Reconfiguring daemon agent.smithi071 on smithi071 2024-03-10T03:51:11.542 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:11 smithi006 ceph-mon[25256]: osdmap e73: 8 total, 8 up, 8 in 2024-03-10T03:51:11.542 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:11 smithi006 ceph-mon[31842]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:51:11.542 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:11 smithi006 ceph-mon[31842]: Reconfiguring daemon agent.smithi006 on smithi006 2024-03-10T03:51:11.542 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:11 smithi006 ceph-mon[31842]: Reconfiguring daemon agent.smithi071 on smithi071 2024-03-10T03:51:11.542 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:11 smithi006 ceph-mon[31842]: osdmap e73: 8 total, 8 up, 8 in 2024-03-10T03:51:11.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:11 smithi071 ceph-mon[27181]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:51:11.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:11 smithi071 ceph-mon[27181]: Reconfiguring daemon agent.smithi006 on smithi006 2024-03-10T03:51:11.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:11 smithi071 ceph-mon[27181]: Reconfiguring daemon agent.smithi071 on smithi071 2024-03-10T03:51:11.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:11 smithi071 ceph-mon[27181]: osdmap e73: 8 total, 8 up, 8 in 2024-03-10T03:51:12.407 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:12 smithi006 ceph-mon[25256]: pgmap v36: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:51:12.407 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:12 smithi006 ceph-mon[25256]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-10T03:51:12.407 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:12 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:12.408 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:12 smithi006 ceph-mon[31842]: pgmap v36: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:51:12.408 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:12 smithi006 ceph-mon[31842]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-10T03:51:12.408 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:12 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:12.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:12 smithi071 ceph-mon[27181]: pgmap v36: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:51:12.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:12 smithi071 ceph-mon[27181]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-10T03:51:12.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:12 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:12.646 INFO:teuthology.orchestra.run.smithi006.stdout:167 167 2024-03-10T03:51:13.086 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:12 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: ::ffff:172.21.15.71 - - [10/Mar/2024:03:51:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-10T03:51:13.298 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- bash -c 'ceph orch status' 2024-03-10T03:51:13.962 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:51:14.292 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:14 smithi006 ceph-mon[25256]: pgmap v37: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-10T03:51:14.292 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:14 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:14.292 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:14 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:14.293 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:14 smithi006 ceph-mon[31842]: pgmap v37: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-10T03:51:14.293 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:14 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:14.293 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:14 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:14.325 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:14 smithi071 ceph-mon[27181]: pgmap v37: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-10T03:51:14.325 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:14 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:14.325 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:14 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:15.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:15 smithi071 ceph-mon[27181]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-10T03:51:15.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:15 smithi071 ceph-mon[27181]: Cluster is now healthy 2024-03-10T03:51:15.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:15 smithi006 ceph-mon[25256]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-10T03:51:15.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:15 smithi006 ceph-mon[25256]: Cluster is now healthy 2024-03-10T03:51:15.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:15 smithi006 ceph-mon[31842]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-10T03:51:15.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:15 smithi006 ceph-mon[31842]: Cluster is now healthy 2024-03-10T03:51:16.450 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:16 smithi006 ceph-mon[25256]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 2 objects/s recovering 2024-03-10T03:51:16.450 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:16 smithi006 ceph-mon[31842]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 2 objects/s recovering 2024-03-10T03:51:16.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:16 smithi071 ceph-mon[27181]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 2 objects/s recovering 2024-03-10T03:51:17.933 INFO:teuthology.orchestra.run.smithi006.stdout:Backend: cephadm 2024-03-10T03:51:17.934 INFO:teuthology.orchestra.run.smithi006.stdout:Available: Yes 2024-03-10T03:51:17.934 INFO:teuthology.orchestra.run.smithi006.stdout:Paused: No 2024-03-10T03:51:18.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:18 smithi006 ceph-mon[25256]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 2 objects/s recovering 2024-03-10T03:51:18.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:18 smithi006 ceph-mon[31842]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 2 objects/s recovering 2024-03-10T03:51:18.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:18 smithi071 ceph-mon[27181]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 2 objects/s recovering 2024-03-10T03:51:18.842 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-10T03:51:19.336 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:19 smithi006 ceph-mon[31842]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:51:19.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:19 smithi006 ceph-mon[25256]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:51:19.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:19 smithi071 ceph-mon[27181]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:51:19.640 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:51:20.091 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:51:19 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a[47087]: debug there is no tcmu-runner data available 2024-03-10T03:51:20.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:20 smithi006 ceph-mon[25256]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 11 B/s, 1 objects/s recovering 2024-03-10T03:51:20.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:20 smithi006 ceph-mon[31842]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 11 B/s, 1 objects/s recovering 2024-03-10T03:51:20.590 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:20 smithi071 ceph-mon[27181]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 11 B/s, 1 objects/s recovering 2024-03-10T03:51:21.381 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:21 smithi006 ceph-mon[25256]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:51:21.381 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:21 smithi006 ceph-mon[31842]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:51:21.590 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:21 smithi071 ceph-mon[27181]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:51:22.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:22 smithi006 ceph-mon[25256]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 997 B/s rd, 0 op/s; 10 B/s, 1 objects/s recovering 2024-03-10T03:51:22.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:22 smithi006 ceph-mon[31842]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 997 B/s rd, 0 op/s; 10 B/s, 1 objects/s recovering 2024-03-10T03:51:22.590 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:22 smithi071 ceph-mon[27181]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 997 B/s rd, 0 op/s; 10 B/s, 1 objects/s recovering 2024-03-10T03:51:22.855 INFO:teuthology.orchestra.run.smithi006.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-10T03:51:22.855 INFO:teuthology.orchestra.run.smithi006.stdout:agent.smithi006 smithi006 starting - - - - 2024-03-10T03:51:22.855 INFO:teuthology.orchestra.run.smithi006.stdout:agent.smithi071 smithi071 starting - - - - 2024-03-10T03:51:22.855 INFO:teuthology.orchestra.run.smithi006.stdout:alertmanager.a smithi006 *:9093,9094 starting - - - - 2024-03-10T03:51:22.855 INFO:teuthology.orchestra.run.smithi006.stdout:grafana.a smithi071 *:3000 starting - - - - 2024-03-10T03:51:22.856 INFO:teuthology.orchestra.run.smithi006.stdout:iscsi.iscsi.a smithi071 *:5000 running (104s) 96s ago 104s - - 3.6 01a4f6a6bb3e ce51f431d58c 2024-03-10T03:51:22.856 INFO:teuthology.orchestra.run.smithi006.stdout:mgr.x smithi071 *:8443,8765 running (5m) 96s ago 5m - - 19.0.0-1875-g831bc188 01a4f6a6bb3e 14276e06b3cb 2024-03-10T03:51:22.856 INFO:teuthology.orchestra.run.smithi006.stdout:mgr.y smithi006 *:9283,8765 running (7m) 93s ago 7m 509M - 19.0.0-1875-g831bc188 01a4f6a6bb3e d42af8aedd6c 2024-03-10T03:51:22.856 INFO:teuthology.orchestra.run.smithi006.stdout:mon.a smithi006 running (7m) 93s ago 7m 43.8M 2048M 19.0.0-1875-g831bc188 01a4f6a6bb3e 95f492a8209d 2024-03-10T03:51:22.856 INFO:teuthology.orchestra.run.smithi006.stdout:mon.b smithi071 running (6m) 96s ago 6m - 2048M 19.0.0-1875-g831bc188 01a4f6a6bb3e e6f6d97fafa3 2024-03-10T03:51:22.856 INFO:teuthology.orchestra.run.smithi006.stdout:mon.c smithi006 running (6m) 93s ago 6m 39.0M 2048M 19.0.0-1875-g831bc188 01a4f6a6bb3e 2f0215995846 2024-03-10T03:51:22.856 INFO:teuthology.orchestra.run.smithi006.stdout:node-exporter.a smithi006 *:9100 starting - - - - 2024-03-10T03:51:22.856 INFO:teuthology.orchestra.run.smithi006.stdout:node-exporter.b smithi071 *:9100 starting - - - - 2024-03-10T03:51:22.856 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0 smithi006 running (5m) 93s ago 5m 48.4M 4096M 19.0.0-1875-g831bc188 01a4f6a6bb3e 932db93cb2f9 2024-03-10T03:51:22.856 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1 smithi006 running (4m) 93s ago 4m 49.1M 4096M 19.0.0-1875-g831bc188 01a4f6a6bb3e 7ab673529cd2 2024-03-10T03:51:22.856 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2 smithi006 running (4m) 93s ago 4m 53.9M 4096M 19.0.0-1875-g831bc188 01a4f6a6bb3e 1b97253c2292 2024-03-10T03:51:22.856 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3 smithi006 running (4m) 93s ago 4m 44.2M 4096M 19.0.0-1875-g831bc188 01a4f6a6bb3e fc197ceb381f 2024-03-10T03:51:22.856 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4 smithi071 running (3m) 96s ago 3m - 4096M 19.0.0-1875-g831bc188 01a4f6a6bb3e 9ea9b48c43d5 2024-03-10T03:51:22.856 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5 smithi071 running (3m) 96s ago 3m - 4096M 19.0.0-1875-g831bc188 01a4f6a6bb3e 979e663e031c 2024-03-10T03:51:22.857 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6 smithi071 running (2m) 96s ago 2m - 4096M 19.0.0-1875-g831bc188 01a4f6a6bb3e c25d08ba979d 2024-03-10T03:51:22.857 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7 smithi071 running (2m) 96s ago 2m - 4096M 19.0.0-1875-g831bc188 01a4f6a6bb3e 0541b9131538 2024-03-10T03:51:22.857 INFO:teuthology.orchestra.run.smithi006.stdout:prometheus.a smithi071 *:9095 starting - - - - 2024-03-10T03:51:22.857 INFO:teuthology.orchestra.run.smithi006.stdout:rgw.foo.a smithi006 *:80 running (119s) 93s ago 118s 75.3M - 19.0.0-1875-g831bc188 01a4f6a6bb3e 379f2bdd26ef 2024-03-10T03:51:23.086 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:22 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: ::ffff:172.21.15.71 - - [10/Mar/2024:03:51:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-10T03:51:23.842 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-10T03:51:24.186 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:51:24.431 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:24 smithi006 ceph-mon[25256]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 9 B/s, 1 objects/s recovering 2024-03-10T03:51:24.431 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:24 smithi006 ceph-mon[25256]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:51:24.431 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:24 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T03:51:24.431 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:24 smithi006 ceph-mon[31842]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 9 B/s, 1 objects/s recovering 2024-03-10T03:51:24.431 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:24 smithi006 ceph-mon[31842]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:51:24.431 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:24 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T03:51:24.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:24 smithi071 ceph-mon[27181]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 9 B/s, 1 objects/s recovering 2024-03-10T03:51:24.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:24 smithi071 ceph-mon[27181]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:51:24.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:24 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T03:51:26.441 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:26 smithi006 ceph-mon[25256]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 9 B/s, 1 objects/s recovering 2024-03-10T03:51:26.441 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:26 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:26.441 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:26 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:26.441 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:26 smithi006 ceph-mon[31842]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 9 B/s, 1 objects/s recovering 2024-03-10T03:51:26.441 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:26 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:26.441 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:26 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:26.486 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:26 smithi071 ceph-mon[27181]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 9 B/s, 1 objects/s recovering 2024-03-10T03:51:26.486 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:26 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:26.487 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:26 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:26.932 INFO:teuthology.orchestra.run.smithi006.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-10T03:51:26.932 INFO:teuthology.orchestra.run.smithi006.stdout:agent 1/2 - 78s * 2024-03-10T03:51:26.932 INFO:teuthology.orchestra.run.smithi006.stdout:alertmanager ?:9093,9094 1/1 1s ago 103s smithi006=a;count:1 2024-03-10T03:51:26.932 INFO:teuthology.orchestra.run.smithi006.stdout:grafana ?:3000 0/1 - 98s smithi071=a;count:1 2024-03-10T03:51:26.932 INFO:teuthology.orchestra.run.smithi006.stdout:iscsi.datapool ?:5000 1/1 100s ago 115s smithi071=iscsi.a;count:1 2024-03-10T03:51:26.932 INFO:teuthology.orchestra.run.smithi006.stdout:mgr 2/2 100s ago 5m smithi006=y;smithi071=x;count:2 2024-03-10T03:51:26.932 INFO:teuthology.orchestra.run.smithi006.stdout:mon 3/3 100s ago 6m smithi006:172.21.15.6=a;smithi006:[v2:172.21.15.6:3301,v1:172.21.15.6:6790]=c;smithi071:172.21.15.71=b;count:3 2024-03-10T03:51:26.933 INFO:teuthology.orchestra.run.smithi006.stdout:node-exporter ?:9100 1/2 - 108s smithi006=a;smithi071=b;count:2 2024-03-10T03:51:26.933 INFO:teuthology.orchestra.run.smithi006.stdout:osd 8 100s ago - 2024-03-10T03:51:26.933 INFO:teuthology.orchestra.run.smithi006.stdout:prometheus ?:9095 0/1 - 112s smithi071=a;count:1 2024-03-10T03:51:26.933 INFO:teuthology.orchestra.run.smithi006.stdout:rgw.foo.a ?:80 1/1 1s ago 2m smithi006=foo.a;count:1 2024-03-10T03:51:27.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:27 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:27.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:27 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:27.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:27 smithi006 ceph-mon[25256]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:51:27.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:27 smithi006 ceph-mon[25256]: from='client.24635 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:51:27.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:27 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:27.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:27 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:27.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:27 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:27.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:27 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:27.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:27 smithi006 ceph-mon[31842]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:51:27.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:27 smithi006 ceph-mon[31842]: from='client.24635 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:51:27.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:27 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:27.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:27 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:27.740 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-10T03:51:27.840 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:27 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:27.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:27 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:27.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:27 smithi071 ceph-mon[27181]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:51:27.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:27 smithi071 ceph-mon[27181]: from='client.24635 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:51:27.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:27 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:27.841 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:27 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:29.027 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:51:30.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:29 smithi006 ceph-mon[25256]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:51:30.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:29 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:30.087 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:29 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:30.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:29 smithi006 ceph-mon[31842]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:51:30.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:29 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:30.087 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:29 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:30.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:29 smithi071 ceph-mon[27181]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:51:30.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:29 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:30.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:29 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:30.091 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:51:29 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a[47087]: debug there is no tcmu-runner data available 2024-03-10T03:51:30.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:30 smithi006 ceph-mon[25256]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:51:30.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:30 smithi006 ceph-mon[31842]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:51:31.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:30 smithi071 ceph-mon[27181]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:51:31.541 INFO:teuthology.orchestra.run.smithi006.stdout:HOST ADDR LABELS STATUS 2024-03-10T03:51:31.541 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 172.21.15.6 2024-03-10T03:51:31.541 INFO:teuthology.orchestra.run.smithi006.stdout:smithi071 172.21.15.71 2024-03-10T03:51:31.541 INFO:teuthology.orchestra.run.smithi006.stdout:2 hosts in cluster 2024-03-10T03:51:31.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:31 smithi006 ceph-mon[25256]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:51:31.836 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:31 smithi006 ceph-mon[31842]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:51:32.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:31 smithi071 ceph-mon[27181]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:51:32.258 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-10T03:51:32.918 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:32 smithi006 ceph-mon[31842]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:51:32.918 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:32 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:32.919 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:32 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:32.919 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:32 smithi006 ceph-mon[25256]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:51:32.919 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:32 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:32.919 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:32 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:32.919 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:32 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: ::ffff:172.21.15.71 - - [10/Mar/2024:03:51:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-10T03:51:32.995 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:51:33.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:32 smithi071 ceph-mon[27181]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:51:33.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:32 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:33.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:32 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:34.042 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:33 smithi006 ceph-mon[25256]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:51:34.042 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:33 smithi006 ceph-mon[25256]: Detected new or changed devices on smithi071 2024-03-10T03:51:34.042 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:33 smithi006 ceph-mon[25256]: Detected new or changed devices on smithi006 2024-03-10T03:51:34.042 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:33 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:34.042 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:33 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:34.042 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:33 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:51:34.042 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:33 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:51:34.042 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:33 smithi006 ceph-mon[25256]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-10T03:51:34.042 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:33 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:34.042 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:33 smithi006 ceph-mon[25256]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-10T03:51:34.043 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:33 smithi006 ceph-mon[25256]: Reconfiguring daemon alertmanager.a on smithi006 2024-03-10T03:51:34.043 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:33 smithi006 ceph-mon[25256]: Health check failed: 2 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-03-10T03:51:34.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:33 smithi006 ceph-mon[31842]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:51:34.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:33 smithi006 ceph-mon[31842]: Detected new or changed devices on smithi071 2024-03-10T03:51:34.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:33 smithi006 ceph-mon[31842]: Detected new or changed devices on smithi006 2024-03-10T03:51:34.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:33 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:34.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:33 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:34.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:33 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:51:34.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:33 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:51:34.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:33 smithi006 ceph-mon[31842]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-10T03:51:34.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:33 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:34.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:33 smithi006 ceph-mon[31842]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-10T03:51:34.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:33 smithi006 ceph-mon[31842]: Reconfiguring daemon alertmanager.a on smithi006 2024-03-10T03:51:34.044 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:33 smithi006 ceph-mon[31842]: Health check failed: 2 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-03-10T03:51:34.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:33 smithi071 ceph-mon[27181]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-10T03:51:34.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:33 smithi071 ceph-mon[27181]: Detected new or changed devices on smithi071 2024-03-10T03:51:34.320 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:33 smithi071 ceph-mon[27181]: Detected new or changed devices on smithi006 2024-03-10T03:51:34.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:33 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:34.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:33 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:34.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:33 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:51:34.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:33 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:51:34.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:33 smithi071 ceph-mon[27181]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-10T03:51:34.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:33 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:34.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:33 smithi071 ceph-mon[27181]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-10T03:51:34.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:33 smithi071 ceph-mon[27181]: Reconfiguring daemon alertmanager.a on smithi006 2024-03-10T03:51:34.321 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:33 smithi071 ceph-mon[27181]: Health check failed: 2 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN) 2024-03-10T03:51:36.306 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:36 smithi071 ceph-mon[27181]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-10T03:51:36.331 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:36 smithi006 ceph-mon[31842]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-10T03:51:36.331 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:36 smithi006 ceph-mon[25256]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-10T03:51:38.241 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:38 smithi006 ceph-mon[25256]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-10T03:51:38.242 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:38 smithi006 ceph-mon[31842]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-10T03:51:38.242 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:38 smithi006 systemd[1]: Stopping Ceph alertmanager.a for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:51:38.341 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:38 smithi071 ceph-mon[27181]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-10T03:51:38.837 INFO:teuthology.orchestra.run.smithi006.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-10T03:51:38.837 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330000H400BGN 372G No 5s ago Has a FileSystem, LVM detected 2024-03-10T03:51:38.837 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/nvme1n1 ssd Linux_9318b97d0f4313945687 89.4G No 5s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-10T03:51:38.837 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/nvme2n1 ssd Linux_ef6dd47f3d99e7414b53 89.4G No 5s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-10T03:51:38.837 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/nvme3n1 ssd Linux_0f26ccc1b1e21586407e 89.4G No 5s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-10T03:51:38.837 INFO:teuthology.orchestra.run.smithi006.stdout:smithi006 /dev/nvme4n1 ssd Linux_244524db6efc1c3ab0d1 89.4G No 5s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-10T03:51:38.837 INFO:teuthology.orchestra.run.smithi006.stdout:smithi071 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62040172400BGN 372G No 6s ago Has a FileSystem, LVM detected 2024-03-10T03:51:38.837 INFO:teuthology.orchestra.run.smithi006.stdout:smithi071 /dev/nvme1n1 ssd Linux_2a52a3098abc918ed187 89.4G No 6s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-10T03:51:38.837 INFO:teuthology.orchestra.run.smithi006.stdout:smithi071 /dev/nvme2n1 ssd Linux_c35632b0b7a7a8491f5b 89.4G No 6s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-10T03:51:38.837 INFO:teuthology.orchestra.run.smithi006.stdout:smithi071 /dev/nvme3n1 ssd Linux_744c282c188a184d7f41 89.4G No 6s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-10T03:51:38.837 INFO:teuthology.orchestra.run.smithi006.stdout:smithi071 /dev/nvme4n1 ssd Linux_db45d12405b7a1353ffb 89.4G No 6s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-10T03:51:39.075 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:38 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[59930]: ts=2024-03-10T03:51:38.805Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-10T03:51:39.075 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:38 smithi006 podman[70116]: 2024-03-10 03:51:38.846260682 +0000 UTC m=+0.171171980 container died 7ae68de3709a4c81af7b837523b6826fa99141616d6ec5bc4b476c27317aee81 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-10T03:51:39.748 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:39 smithi006 podman[70116]: 2024-03-10 03:51:39.339594375 +0000 UTC m=+0.664505662 container cleanup 7ae68de3709a4c81af7b837523b6826fa99141616d6ec5bc4b476c27317aee81 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-10T03:51:39.748 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:39 smithi006 bash[70116]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a 2024-03-10T03:51:40.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:39 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:40.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:39 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T03:51:40.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:39 smithi006 ceph-mon[25256]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:51:40.046 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:39 smithi006 ceph-mon[25256]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-10T03:51:40.046 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:39 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:40.046 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:39 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T03:51:40.046 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:39 smithi006 ceph-mon[31842]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:51:40.046 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:39 smithi006 ceph-mon[31842]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-10T03:51:40.046 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:39 smithi006 podman[70132]: 2024-03-10 03:51:39.792198816 +0000 UTC m=+0.937317992 container remove 7ae68de3709a4c81af7b837523b6826fa99141616d6ec5bc4b476c27317aee81 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-10T03:51:40.046 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:39 smithi006 podman[70132]: 2024-03-10 03:51:39.876045607 +0000 UTC m=+1.021164793 volume remove 7e1982548e5c4966fdb8913315094fe094e388d785df41b92d41c9c4e47b2933 2024-03-10T03:51:40.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:39 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:40.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:39 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-10T03:51:40.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:39 smithi071 ceph-mon[27181]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-10T03:51:40.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:39 smithi071 ceph-mon[27181]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-10T03:51:40.091 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:51:39 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a[47087]: debug there is no tcmu-runner data available 2024-03-10T03:51:40.336 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:40 smithi006 systemd[1]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-10T03:51:40.336 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:40 smithi006 systemd[1]: Stopped Ceph alertmanager.a for 414ba1e0-de90-11ee-95c7-87774f69a715. 2024-03-10T03:51:40.336 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:40 smithi006 systemd[1]: Starting Ceph alertmanager.a for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:51:40.497 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-03-10T03:51:40.719 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:40 smithi006 podman[70257]: 2024-03-10 03:51:40.607341823 +0000 UTC m=+0.018190734 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-10T03:51:40.719 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:40 smithi006 podman[70257]: 2024-03-10 03:51:40.719498351 +0000 UTC m=+0.130347246 volume create 65de3704eda0f3f9daf2f66a88648764865711e7f92a1b132026912593369491 2024-03-10T03:51:40.996 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:40 smithi006 ceph-mon[25256]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:51:40.996 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:40 smithi006 ceph-mon[31842]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:51:40.996 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:40 smithi006 podman[70257]: 2024-03-10 03:51:40.803287091 +0000 UTC m=+0.214135982 container create 1b85b458da5b874b97d990604127deb2444303c1e0471ba5b6e8e1b9942bae07 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-10T03:51:41.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:40 smithi071 ceph-mon[27181]: from='client.24427 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-10T03:51:41.300 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:51:41.337 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:41 smithi006 podman[70257]: 2024-03-10 03:51:41.08086476 +0000 UTC m=+0.491713666 container init 1b85b458da5b874b97d990604127deb2444303c1e0471ba5b6e8e1b9942bae07 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-10T03:51:41.337 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:41 smithi006 podman[70257]: 2024-03-10 03:51:41.083509487 +0000 UTC m=+0.494358387 container start 1b85b458da5b874b97d990604127deb2444303c1e0471ba5b6e8e1b9942bae07 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-10T03:51:41.337 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:41 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[70283]: ts=2024-03-10T03:51:41.095Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-10T03:51:41.337 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:41 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[70283]: ts=2024-03-10T03:51:41.095Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-10T03:51:41.337 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:41 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[70283]: ts=2024-03-10T03:51:41.096Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.6 port=9094 2024-03-10T03:51:41.337 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:41 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[70283]: ts=2024-03-10T03:51:41.097Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-10T03:51:41.337 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:41 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[70283]: ts=2024-03-10T03:51:41.121Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-10T03:51:41.337 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:41 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[70283]: ts=2024-03-10T03:51:41.121Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-10T03:51:41.337 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:41 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[70283]: ts=2024-03-10T03:51:41.124Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-10T03:51:41.337 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:41 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[70283]: ts=2024-03-10T03:51:41.124Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-10T03:51:41.337 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:41 smithi006 bash[70257]: 1b85b458da5b874b97d990604127deb2444303c1e0471ba5b6e8e1b9942bae07 2024-03-10T03:51:41.337 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:41 smithi006 systemd[1]: Started Ceph alertmanager.a for 414ba1e0-de90-11ee-95c7-87774f69a715. 2024-03-10T03:51:42.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:42 smithi006 ceph-mon[25256]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-10T03:51:42.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:42 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:42.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:42 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:42.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:42 smithi006 ceph-mon[25256]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-10T03:51:42.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:42 smithi006 ceph-mon[31842]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-10T03:51:42.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:42 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:42.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:42 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:42.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:42 smithi006 ceph-mon[31842]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-10T03:51:42.369 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:42 smithi071 ceph-mon[27181]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-10T03:51:42.369 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:42 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:42.369 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:42 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:42.370 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:42 smithi071 ceph-mon[27181]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-10T03:51:43.086 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:42 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: ::ffff:172.21.15.71 - - [10/Mar/2024:03:51:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-10T03:51:43.439 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:43 smithi006 ceph-mon[25256]: Reconfiguring daemon prometheus.a on smithi071 2024-03-10T03:51:43.439 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:43 smithi006 ceph-mon[31842]: Reconfiguring daemon prometheus.a on smithi071 2024-03-10T03:51:43.439 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:43 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[70283]: ts=2024-03-10T03:51:43.097Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000101264s 2024-03-10T03:51:43.477 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:43 smithi071 ceph-mon[27181]: Reconfiguring daemon prometheus.a on smithi071 2024-03-10T03:51:43.784 INFO:teuthology.orchestra.run.smithi006.stdout:service_type: agent 2024-03-10T03:51:43.784 INFO:teuthology.orchestra.run.smithi006.stdout:service_name: agent 2024-03-10T03:51:43.785 INFO:teuthology.orchestra.run.smithi006.stdout:placement: 2024-03-10T03:51:43.785 INFO:teuthology.orchestra.run.smithi006.stdout: host_pattern: '*' 2024-03-10T03:51:43.785 INFO:teuthology.orchestra.run.smithi006.stdout:status: 2024-03-10T03:51:43.785 INFO:teuthology.orchestra.run.smithi006.stdout: created: '2024-03-10T03:50:08.613805Z' 2024-03-10T03:51:43.785 INFO:teuthology.orchestra.run.smithi006.stdout: last_refresh: '2024-03-10T03:51:25.171267Z' 2024-03-10T03:51:43.785 INFO:teuthology.orchestra.run.smithi006.stdout: running: 0 2024-03-10T03:51:43.785 INFO:teuthology.orchestra.run.smithi006.stdout: size: 2 2024-03-10T03:51:43.785 INFO:teuthology.orchestra.run.smithi006.stdout:events: 2024-03-10T03:51:43.785 INFO:teuthology.orchestra.run.smithi006.stdout:- 2024-03-10T03:50:10.033357Z service:agent [INFO] "service was created" 2024-03-10T03:51:43.785 INFO:teuthology.orchestra.run.smithi006.stdout:--- 2024-03-10T03:51:43.785 INFO:teuthology.orchestra.run.smithi006.stdout:service_type: alertmanager 2024-03-10T03:51:43.785 INFO:teuthology.orchestra.run.smithi006.stdout:service_name: alertmanager 2024-03-10T03:51:43.786 INFO:teuthology.orchestra.run.smithi006.stdout:placement: 2024-03-10T03:51:43.786 INFO:teuthology.orchestra.run.smithi006.stdout: count: 1 2024-03-10T03:51:43.786 INFO:teuthology.orchestra.run.smithi006.stdout: hosts: 2024-03-10T03:51:43.786 INFO:teuthology.orchestra.run.smithi006.stdout: - smithi006=a 2024-03-10T03:51:43.786 INFO:teuthology.orchestra.run.smithi006.stdout:status: 2024-03-10T03:51:43.786 INFO:teuthology.orchestra.run.smithi006.stdout: created: '2024-03-10T03:49:43.925485Z' 2024-03-10T03:51:43.786 INFO:teuthology.orchestra.run.smithi006.stdout: ports: 2024-03-10T03:51:43.786 INFO:teuthology.orchestra.run.smithi006.stdout: - 9093 2024-03-10T03:51:43.786 INFO:teuthology.orchestra.run.smithi006.stdout: - 9094 2024-03-10T03:51:43.786 INFO:teuthology.orchestra.run.smithi006.stdout: running: 0 2024-03-10T03:51:43.786 INFO:teuthology.orchestra.run.smithi006.stdout: size: 1 2024-03-10T03:51:43.787 INFO:teuthology.orchestra.run.smithi006.stdout:events: 2024-03-10T03:51:43.787 INFO:teuthology.orchestra.run.smithi006.stdout:- 2024-03-10T03:50:43.897496Z service:alertmanager [INFO] "service was created" 2024-03-10T03:51:43.787 INFO:teuthology.orchestra.run.smithi006.stdout:--- 2024-03-10T03:51:43.787 INFO:teuthology.orchestra.run.smithi006.stdout:service_type: grafana 2024-03-10T03:51:43.787 INFO:teuthology.orchestra.run.smithi006.stdout:service_name: grafana 2024-03-10T03:51:43.787 INFO:teuthology.orchestra.run.smithi006.stdout:placement: 2024-03-10T03:51:43.787 INFO:teuthology.orchestra.run.smithi006.stdout: count: 1 2024-03-10T03:51:43.787 INFO:teuthology.orchestra.run.smithi006.stdout: hosts: 2024-03-10T03:51:43.787 INFO:teuthology.orchestra.run.smithi006.stdout: - smithi071=a 2024-03-10T03:51:43.787 INFO:teuthology.orchestra.run.smithi006.stdout:spec: 2024-03-10T03:51:43.787 INFO:teuthology.orchestra.run.smithi006.stdout: anonymous_access: true 2024-03-10T03:51:43.787 INFO:teuthology.orchestra.run.smithi006.stdout: protocol: https 2024-03-10T03:51:43.787 INFO:teuthology.orchestra.run.smithi006.stdout:status: 2024-03-10T03:51:43.787 INFO:teuthology.orchestra.run.smithi006.stdout: created: '2024-03-10T03:49:47.984609Z' 2024-03-10T03:51:43.787 INFO:teuthology.orchestra.run.smithi006.stdout: last_refresh: '2024-03-10T03:51:27.472864Z' 2024-03-10T03:51:43.788 INFO:teuthology.orchestra.run.smithi006.stdout: ports: 2024-03-10T03:51:43.788 INFO:teuthology.orchestra.run.smithi006.stdout: - 3000 2024-03-10T03:51:43.788 INFO:teuthology.orchestra.run.smithi006.stdout: running: 1 2024-03-10T03:51:43.788 INFO:teuthology.orchestra.run.smithi006.stdout: size: 1 2024-03-10T03:51:43.788 INFO:teuthology.orchestra.run.smithi006.stdout:events: 2024-03-10T03:51:43.788 INFO:teuthology.orchestra.run.smithi006.stdout:- 2024-03-10T03:50:58.308015Z service:grafana [INFO] "service was created" 2024-03-10T03:51:43.788 INFO:teuthology.orchestra.run.smithi006.stdout:--- 2024-03-10T03:51:43.788 INFO:teuthology.orchestra.run.smithi006.stdout:service_type: iscsi 2024-03-10T03:51:43.788 INFO:teuthology.orchestra.run.smithi006.stdout:service_id: datapool 2024-03-10T03:51:43.788 INFO:teuthology.orchestra.run.smithi006.stdout:service_name: iscsi.datapool 2024-03-10T03:51:43.788 INFO:teuthology.orchestra.run.smithi006.stdout:placement: 2024-03-10T03:51:43.788 INFO:teuthology.orchestra.run.smithi006.stdout: count: 1 2024-03-10T03:51:43.788 INFO:teuthology.orchestra.run.smithi006.stdout: hosts: 2024-03-10T03:51:43.788 INFO:teuthology.orchestra.run.smithi006.stdout: - smithi071=iscsi.a 2024-03-10T03:51:43.788 INFO:teuthology.orchestra.run.smithi006.stdout:spec: 2024-03-10T03:51:43.789 INFO:teuthology.orchestra.run.smithi006.stdout: api_password: admin 2024-03-10T03:51:43.789 INFO:teuthology.orchestra.run.smithi006.stdout: api_port: 5000 2024-03-10T03:51:43.789 INFO:teuthology.orchestra.run.smithi006.stdout: api_user: admin 2024-03-10T03:51:43.789 INFO:teuthology.orchestra.run.smithi006.stdout: pool: datapool 2024-03-10T03:51:43.789 INFO:teuthology.orchestra.run.smithi006.stdout: trusted_ip_list: 172.21.15.71 2024-03-10T03:51:43.789 INFO:teuthology.orchestra.run.smithi006.stdout:status: 2024-03-10T03:51:43.789 INFO:teuthology.orchestra.run.smithi006.stdout: created: '2024-03-10T03:49:31.915144Z' 2024-03-10T03:51:43.789 INFO:teuthology.orchestra.run.smithi006.stdout: last_refresh: '2024-03-10T03:51:27.472713Z' 2024-03-10T03:51:43.789 INFO:teuthology.orchestra.run.smithi006.stdout: ports: 2024-03-10T03:51:43.789 INFO:teuthology.orchestra.run.smithi006.stdout: - 5000 2024-03-10T03:51:43.789 INFO:teuthology.orchestra.run.smithi006.stdout: running: 1 2024-03-10T03:51:43.789 INFO:teuthology.orchestra.run.smithi006.stdout: size: 1 2024-03-10T03:51:43.789 INFO:teuthology.orchestra.run.smithi006.stdout:--- 2024-03-10T03:51:43.789 INFO:teuthology.orchestra.run.smithi006.stdout:service_type: mgr 2024-03-10T03:51:43.789 INFO:teuthology.orchestra.run.smithi006.stdout:service_name: mgr 2024-03-10T03:51:43.789 INFO:teuthology.orchestra.run.smithi006.stdout:placement: 2024-03-10T03:51:43.790 INFO:teuthology.orchestra.run.smithi006.stdout: count: 2 2024-03-10T03:51:43.790 INFO:teuthology.orchestra.run.smithi006.stdout: hosts: 2024-03-10T03:51:43.790 INFO:teuthology.orchestra.run.smithi006.stdout: - smithi006=y 2024-03-10T03:51:43.790 INFO:teuthology.orchestra.run.smithi006.stdout: - smithi071=x 2024-03-10T03:51:43.790 INFO:teuthology.orchestra.run.smithi006.stdout:status: 2024-03-10T03:51:43.790 INFO:teuthology.orchestra.run.smithi006.stdout: created: '2024-03-10T03:45:35.635470Z' 2024-03-10T03:51:43.790 INFO:teuthology.orchestra.run.smithi006.stdout: last_refresh: '2024-03-10T03:51:25.171203Z' 2024-03-10T03:51:43.790 INFO:teuthology.orchestra.run.smithi006.stdout: running: 2 2024-03-10T03:51:43.790 INFO:teuthology.orchestra.run.smithi006.stdout: size: 2 2024-03-10T03:51:43.790 INFO:teuthology.orchestra.run.smithi006.stdout:--- 2024-03-10T03:51:43.790 INFO:teuthology.orchestra.run.smithi006.stdout:service_type: mon 2024-03-10T03:51:43.790 INFO:teuthology.orchestra.run.smithi006.stdout:service_name: mon 2024-03-10T03:51:43.790 INFO:teuthology.orchestra.run.smithi006.stdout:placement: 2024-03-10T03:51:43.790 INFO:teuthology.orchestra.run.smithi006.stdout: count: 3 2024-03-10T03:51:43.790 INFO:teuthology.orchestra.run.smithi006.stdout: hosts: 2024-03-10T03:51:43.791 INFO:teuthology.orchestra.run.smithi006.stdout: - smithi006:172.21.15.6=a 2024-03-10T03:51:43.791 INFO:teuthology.orchestra.run.smithi006.stdout: - smithi006:[v2:172.21.15.6:3301,v1:172.21.15.6:6790]=c 2024-03-10T03:51:43.791 INFO:teuthology.orchestra.run.smithi006.stdout: - smithi071:172.21.15.71=b 2024-03-10T03:51:43.791 INFO:teuthology.orchestra.run.smithi006.stdout:status: 2024-03-10T03:51:43.791 INFO:teuthology.orchestra.run.smithi006.stdout: created: '2024-03-10T03:45:04.767222Z' 2024-03-10T03:51:43.791 INFO:teuthology.orchestra.run.smithi006.stdout: last_refresh: '2024-03-10T03:51:25.171089Z' 2024-03-10T03:51:43.791 INFO:teuthology.orchestra.run.smithi006.stdout: running: 3 2024-03-10T03:51:43.791 INFO:teuthology.orchestra.run.smithi006.stdout: size: 3 2024-03-10T03:51:43.791 INFO:teuthology.orchestra.run.smithi006.stdout:--- 2024-03-10T03:51:43.791 INFO:teuthology.orchestra.run.smithi006.stdout:service_type: node-exporter 2024-03-10T03:51:43.791 INFO:teuthology.orchestra.run.smithi006.stdout:service_name: node-exporter 2024-03-10T03:51:43.791 INFO:teuthology.orchestra.run.smithi006.stdout:placement: 2024-03-10T03:51:43.791 INFO:teuthology.orchestra.run.smithi006.stdout: count: 2 2024-03-10T03:51:43.791 INFO:teuthology.orchestra.run.smithi006.stdout: hosts: 2024-03-10T03:51:43.791 INFO:teuthology.orchestra.run.smithi006.stdout: - smithi006=a 2024-03-10T03:51:43.791 INFO:teuthology.orchestra.run.smithi006.stdout: - smithi071=b 2024-03-10T03:51:43.792 INFO:teuthology.orchestra.run.smithi006.stdout:status: 2024-03-10T03:51:43.792 INFO:teuthology.orchestra.run.smithi006.stdout: created: '2024-03-10T03:49:38.422981Z' 2024-03-10T03:51:43.792 INFO:teuthology.orchestra.run.smithi006.stdout: last_refresh: '2024-03-10T03:51:25.171943Z' 2024-03-10T03:51:43.792 INFO:teuthology.orchestra.run.smithi006.stdout: ports: 2024-03-10T03:51:43.792 INFO:teuthology.orchestra.run.smithi006.stdout: - 9100 2024-03-10T03:51:43.792 INFO:teuthology.orchestra.run.smithi006.stdout: running: 2 2024-03-10T03:51:43.792 INFO:teuthology.orchestra.run.smithi006.stdout: size: 2 2024-03-10T03:51:43.792 INFO:teuthology.orchestra.run.smithi006.stdout:events: 2024-03-10T03:51:43.792 INFO:teuthology.orchestra.run.smithi006.stdout:- 2024-03-10T03:51:09.227685Z service:node-exporter [INFO] "service was created" 2024-03-10T03:51:43.792 INFO:teuthology.orchestra.run.smithi006.stdout:--- 2024-03-10T03:51:43.792 INFO:teuthology.orchestra.run.smithi006.stdout:service_type: osd 2024-03-10T03:51:43.792 INFO:teuthology.orchestra.run.smithi006.stdout:service_name: osd 2024-03-10T03:51:43.792 INFO:teuthology.orchestra.run.smithi006.stdout:unmanaged: true 2024-03-10T03:51:43.792 INFO:teuthology.orchestra.run.smithi006.stdout:spec: 2024-03-10T03:51:43.792 INFO:teuthology.orchestra.run.smithi006.stdout: filter_logic: AND 2024-03-10T03:51:43.792 INFO:teuthology.orchestra.run.smithi006.stdout: objectstore: bluestore 2024-03-10T03:51:43.793 INFO:teuthology.orchestra.run.smithi006.stdout:status: 2024-03-10T03:51:43.793 INFO:teuthology.orchestra.run.smithi006.stdout: container_image_id: 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 2024-03-10T03:51:43.793 INFO:teuthology.orchestra.run.smithi006.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada 2024-03-10T03:51:43.793 INFO:teuthology.orchestra.run.smithi006.stdout: last_refresh: '2024-03-10T03:51:25.171440Z' 2024-03-10T03:51:43.793 INFO:teuthology.orchestra.run.smithi006.stdout: running: 8 2024-03-10T03:51:43.793 INFO:teuthology.orchestra.run.smithi006.stdout: size: 8 2024-03-10T03:51:43.793 INFO:teuthology.orchestra.run.smithi006.stdout:--- 2024-03-10T03:51:43.793 INFO:teuthology.orchestra.run.smithi006.stdout:service_type: prometheus 2024-03-10T03:51:43.793 INFO:teuthology.orchestra.run.smithi006.stdout:service_name: prometheus 2024-03-10T03:51:43.793 INFO:teuthology.orchestra.run.smithi006.stdout:placement: 2024-03-10T03:51:43.793 INFO:teuthology.orchestra.run.smithi006.stdout: count: 1 2024-03-10T03:51:43.794 INFO:teuthology.orchestra.run.smithi006.stdout: hosts: 2024-03-10T03:51:43.794 INFO:teuthology.orchestra.run.smithi006.stdout: - smithi071=a 2024-03-10T03:51:43.794 INFO:teuthology.orchestra.run.smithi006.stdout:status: 2024-03-10T03:51:43.794 INFO:teuthology.orchestra.run.smithi006.stdout: created: '2024-03-10T03:49:34.820956Z' 2024-03-10T03:51:43.794 INFO:teuthology.orchestra.run.smithi006.stdout: last_refresh: '2024-03-10T03:51:27.472789Z' 2024-03-10T03:51:43.794 INFO:teuthology.orchestra.run.smithi006.stdout: ports: 2024-03-10T03:51:43.794 INFO:teuthology.orchestra.run.smithi006.stdout: - 9095 2024-03-10T03:51:43.794 INFO:teuthology.orchestra.run.smithi006.stdout: running: 1 2024-03-10T03:51:43.794 INFO:teuthology.orchestra.run.smithi006.stdout: size: 1 2024-03-10T03:51:43.794 INFO:teuthology.orchestra.run.smithi006.stdout:events: 2024-03-10T03:51:43.794 INFO:teuthology.orchestra.run.smithi006.stdout:- 2024-03-10T03:51:09.236868Z service:prometheus [INFO] "service was created" 2024-03-10T03:51:43.794 INFO:teuthology.orchestra.run.smithi006.stdout:--- 2024-03-10T03:51:43.794 INFO:teuthology.orchestra.run.smithi006.stdout:service_type: rgw 2024-03-10T03:51:43.794 INFO:teuthology.orchestra.run.smithi006.stdout:service_id: foo.a 2024-03-10T03:51:43.795 INFO:teuthology.orchestra.run.smithi006.stdout:service_name: rgw.foo.a 2024-03-10T03:51:43.795 INFO:teuthology.orchestra.run.smithi006.stdout:placement: 2024-03-10T03:51:43.795 INFO:teuthology.orchestra.run.smithi006.stdout: count: 1 2024-03-10T03:51:43.795 INFO:teuthology.orchestra.run.smithi006.stdout: hosts: 2024-03-10T03:51:43.795 INFO:teuthology.orchestra.run.smithi006.stdout: - smithi006=foo.a 2024-03-10T03:51:43.795 INFO:teuthology.orchestra.run.smithi006.stdout:status: 2024-03-10T03:51:43.795 INFO:teuthology.orchestra.run.smithi006.stdout: created: '2024-03-10T03:49:24.225238Z' 2024-03-10T03:51:43.795 INFO:teuthology.orchestra.run.smithi006.stdout: last_refresh: '2024-03-10T03:51:25.171777Z' 2024-03-10T03:51:43.795 INFO:teuthology.orchestra.run.smithi006.stdout: ports: 2024-03-10T03:51:43.795 INFO:teuthology.orchestra.run.smithi006.stdout: - 80 2024-03-10T03:51:43.795 INFO:teuthology.orchestra.run.smithi006.stdout: running: 1 2024-03-10T03:51:43.795 INFO:teuthology.orchestra.run.smithi006.stdout: size: 1 2024-03-10T03:51:44.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:44 smithi006 ceph-mon[25256]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-10T03:51:44.337 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:44 smithi006 ceph-mon[31842]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-10T03:51:44.517 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:44 smithi071 ceph-mon[27181]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-10T03:51:44.725 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-10T03:51:44.811 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:44 smithi071 systemd[1]: Stopping Ceph prometheus.a for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:51:44.977 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/mon.a/config 2024-03-10T03:51:45.091 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:51:44.922Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-10T03:51:45.091 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:51:44.922Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-10T03:51:45.091 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:51:44.922Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-10T03:51:45.091 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:51:44.922Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-10T03:51:45.091 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:51:44.922Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-10T03:51:45.091 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:51:44.922Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-10T03:51:45.091 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:51:44.922Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-10T03:51:45.091 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:51:44.922Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-10T03:51:45.092 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:51:44.923Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-10T03:51:45.092 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:51:44.923Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-10T03:51:45.092 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:51:44.923Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-10T03:51:45.092 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[48849]: ts=2024-03-10T03:51:44.923Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-10T03:51:45.092 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:44 smithi071 podman[59934]: 2024-03-10 03:51:44.932393593 +0000 UTC m=+0.139306561 container died 008f9b15c15b3c75925af7fe13a177d461d4784d9619a250bc009158233ddf6d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-10T03:51:45.464 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:45 smithi006 ceph-mon[25256]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-10T03:51:45.465 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:45 smithi006 ceph-mon[31842]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-10T03:51:45.556 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:45 smithi071 podman[59934]: 2024-03-10 03:51:45.32084336 +0000 UTC m=+0.527756326 container cleanup 008f9b15c15b3c75925af7fe13a177d461d4784d9619a250bc009158233ddf6d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-10T03:51:45.556 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:45 smithi071 bash[59934]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a 2024-03-10T03:51:45.556 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:45 smithi071 ceph-mon[27181]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-10T03:51:45.840 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:45 smithi071 podman[59949]: 2024-03-10 03:51:45.555684867 +0000 UTC m=+0.623879567 container remove 008f9b15c15b3c75925af7fe13a177d461d4784d9619a250bc009158233ddf6d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-10T03:51:45.841 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:45 smithi071 systemd[1]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-10T03:51:45.841 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:45 smithi071 systemd[1]: Stopped Ceph prometheus.a for 414ba1e0-de90-11ee-95c7-87774f69a715. 2024-03-10T03:51:45.841 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:45 smithi071 systemd[1]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715@prometheus.a.service: Consumed 1.210s CPU time. 2024-03-10T03:51:45.841 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:45 smithi071 systemd[1]: Starting Ceph prometheus.a for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:51:46.200 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:45 smithi071 podman[60056]: 2024-03-10 03:51:45.821870773 +0000 UTC m=+0.018909318 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-10T03:51:46.201 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:45 smithi071 podman[60056]: 2024-03-10 03:51:45.973484746 +0000 UTC m=+0.170523283 container create 1b2d5f2bc0817a3227b106a1cff99e392a034a9d497b555145d4d9d8184e0421 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-10T03:51:46.469 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:46 smithi006 ceph-mon[25256]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:51:46.470 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:46 smithi006 ceph-mon[31842]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:51:46.470 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:51:46] ENGINE Bus STOPPING 2024-03-10T03:51:46.470 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:51:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-10T03:51:46.470 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:51:46] ENGINE Bus STOPPED 2024-03-10T03:51:46.470 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:51:46] ENGINE Bus STARTING 2024-03-10T03:51:46.470 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:51:46] ENGINE Serving on http://:::9283 2024-03-10T03:51:46.470 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:51:46] ENGINE Bus STARTED 2024-03-10T03:51:46.590 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:46 smithi071 ceph-mon[27181]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-10T03:51:46.591 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 podman[60056]: 2024-03-10 03:51:46.200574133 +0000 UTC m=+0.397612681 container init 1b2d5f2bc0817a3227b106a1cff99e392a034a9d497b555145d4d9d8184e0421 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-10T03:51:46.591 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 podman[60056]: 2024-03-10 03:51:46.203406428 +0000 UTC m=+0.400444974 container start 1b2d5f2bc0817a3227b106a1cff99e392a034a9d497b555145d4d9d8184e0421 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-10T03:51:46.591 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:51:46.226Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-10T03:51:46.591 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:51:46.226Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-10T03:51:46.591 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:51:46.226Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi071 (none))" 2024-03-10T03:51:46.592 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:51:46.226Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-10T03:51:46.592 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:51:46.226Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-10T03:51:46.592 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:51:46.228Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-10T03:51:46.592 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:51:46.228Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-10T03:51:46.592 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:51:46.229Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-10T03:51:46.592 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:51:46.229Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-10T03:51:46.592 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:51:46.232Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-10T03:51:46.592 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:51:46.232Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.5µs 2024-03-10T03:51:46.592 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:51:46.232Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-10T03:51:46.592 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:51:46.233Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-10T03:51:46.592 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:51:46.233Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-10T03:51:46.592 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:51:46.233Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=28.826µs wal_replay_duration=804.437µs wbl_replay_duration=102ns total_replay_duration=864.432µs 2024-03-10T03:51:46.592 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:51:46.234Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-10T03:51:46.592 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:51:46.234Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-10T03:51:46.593 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:51:46.234Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-10T03:51:46.593 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:51:46.254Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.853236ms db_storage=714ns remote_storage=996ns web_handler=336ns query_engine=532ns scrape=376.669µs scrape_sd=96.641µs notify=16.236µs notify_sd=14.412µs rules=19.085868ms tracing=5.854µs 2024-03-10T03:51:46.593 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:51:46.254Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-10T03:51:46.593 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:51:46.254Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-10T03:51:46.593 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 bash[60056]: 1b2d5f2bc0817a3227b106a1cff99e392a034a9d497b555145d4d9d8184e0421 2024-03-10T03:51:46.593 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:51:46 smithi071 systemd[1]: Started Ceph prometheus.a for 414ba1e0-de90-11ee-95c7-87774f69a715. 2024-03-10T03:51:46.837 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:51:46] ENGINE Bus STOPPING 2024-03-10T03:51:46.837 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:51:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-10T03:51:46.837 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:51:46] ENGINE Bus STOPPED 2024-03-10T03:51:46.837 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:51:46] ENGINE Bus STARTING 2024-03-10T03:51:46.837 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:51:46] ENGINE Serving on http://:::9283 2024-03-10T03:51:46.837 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:51:46] ENGINE Bus STARTED 2024-03-10T03:51:46.837 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:51:46] ENGINE Bus STOPPING 2024-03-10T03:51:46.837 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:51:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-10T03:51:46.837 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:51:46] ENGINE Bus STOPPED 2024-03-10T03:51:46.837 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:51:46] ENGINE Bus STARTING 2024-03-10T03:51:46.837 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:51:46] ENGINE Serving on http://:::9283 2024-03-10T03:51:46.837 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:46 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:51:46] ENGINE Bus STARTED 2024-03-10T03:51:46.935 INFO:teuthology.orchestra.run.smithi006.stdout:osd 8 21s ago - 2024-03-10T03:51:47.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:47 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:47.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:47 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:47.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:47 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-10T03:51:47.569 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:47 smithi071 ceph-mon[27181]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-10T03:51:47.570 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:47 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi006.front.sepia.ceph.com:9093"}]: dispatch 2024-03-10T03:51:47.570 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:47 smithi071 ceph-mon[27181]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi006.front.sepia.ceph.com:9093"}]: dispatch 2024-03-10T03:51:47.570 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:47 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:47.570 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:47 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-10T03:51:47.570 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:47 smithi071 ceph-mon[27181]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-10T03:51:47.570 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:47 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi071.front.sepia.ceph.com:9095"}]: dispatch 2024-03-10T03:51:47.570 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:47 smithi071 ceph-mon[27181]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi071.front.sepia.ceph.com:9095"}]: dispatch 2024-03-10T03:51:47.570 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:47 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:47.570 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:47 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-10T03:51:47.570 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:47 smithi071 ceph-mon[27181]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-10T03:51:47.570 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:47 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi071.front.sepia.ceph.com:3000"}]: dispatch 2024-03-10T03:51:47.570 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:47 smithi071 ceph-mon[27181]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi071.front.sepia.ceph.com:3000"}]: dispatch 2024-03-10T03:51:47.570 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:47 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:47.570 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:47 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:51:47.571 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:47 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:51:47.571 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:47 smithi071 ceph-mon[27181]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:51:47.571 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:47 smithi071 ceph-mon[27181]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 897 B/s rd, 0 op/s 2024-03-10T03:51:47.571 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:47 smithi071 ceph-mon[27181]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:47.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:47.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:47.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-10T03:51:47.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[25256]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-10T03:51:47.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi006.front.sepia.ceph.com:9093"}]: dispatch 2024-03-10T03:51:47.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[25256]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi006.front.sepia.ceph.com:9093"}]: dispatch 2024-03-10T03:51:47.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:47.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-10T03:51:47.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[25256]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-10T03:51:47.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi071.front.sepia.ceph.com:9095"}]: dispatch 2024-03-10T03:51:47.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[25256]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi071.front.sepia.ceph.com:9095"}]: dispatch 2024-03-10T03:51:47.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:47.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-10T03:51:47.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[25256]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-10T03:51:47.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi071.front.sepia.ceph.com:3000"}]: dispatch 2024-03-10T03:51:47.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[25256]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi071.front.sepia.ceph.com:3000"}]: dispatch 2024-03-10T03:51:47.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:47.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:51:47.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:51:47.589 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[25256]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:51:47.589 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[25256]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 897 B/s rd, 0 op/s 2024-03-10T03:51:47.589 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[25256]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:47.589 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:47.589 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:47.589 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-10T03:51:47.589 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[31842]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-10T03:51:47.589 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi006.front.sepia.ceph.com:9093"}]: dispatch 2024-03-10T03:51:47.589 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[31842]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi006.front.sepia.ceph.com:9093"}]: dispatch 2024-03-10T03:51:47.589 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:47.589 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-10T03:51:47.589 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[31842]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-10T03:51:47.589 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi071.front.sepia.ceph.com:9095"}]: dispatch 2024-03-10T03:51:47.590 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[31842]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi071.front.sepia.ceph.com:9095"}]: dispatch 2024-03-10T03:51:47.590 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:47.590 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-10T03:51:47.590 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[31842]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-10T03:51:47.590 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi071.front.sepia.ceph.com:3000"}]: dispatch 2024-03-10T03:51:47.590 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[31842]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi071.front.sepia.ceph.com:3000"}]: dispatch 2024-03-10T03:51:47.590 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:47.590 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-10T03:51:47.590 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-10T03:51:47.590 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[31842]: from='mgr.24467 172.21.15.6:0/1538346613' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-10T03:51:47.590 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[31842]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 897 B/s rd, 0 op/s 2024-03-10T03:51:47.591 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:47 smithi006 ceph-mon[31842]: from='mgr.24467 ' entity='mgr.y' 2024-03-10T03:51:47.715 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-10T03:51:47.724 INFO:tasks.cephadm:Teardown begin 2024-03-10T03:51:47.725 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-10T03:51:47.756 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-10T03:51:47.788 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-10T03:51:47.789 DEBUG:teuthology.orchestra.run.smithi006:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-10T03:51:47.811 DEBUG:teuthology.orchestra.run.smithi071:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-10T03:51:47.844 INFO:tasks.cephadm:Stopping all daemons... 2024-03-10T03:51:47.844 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-10T03:51:47.845 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mon.a 2024-03-10T03:51:48.086 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:47 smithi006 systemd[1]: Stopping Ceph mon.a for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:51:48.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:48 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-a[25252]: 2024-03-10T03:51:48.136+0000 7fe65c611700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T03:51:48.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:48 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-a[25252]: 2024-03-10T03:51:48.136+0000 7fe65c611700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-10T03:51:48.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:48 smithi006 podman[70839]: 2024-03-10 03:51:48.301452241 +0000 UTC m=+0.265497952 container died 95f492a8209d21bc1066387178a63d18d00bc022f85e2ab0cca33b5359771c1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-10T03:51:49.014 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:48 smithi006 podman[70839]: 2024-03-10 03:51:48.771219362 +0000 UTC m=+0.735265082 container cleanup 95f492a8209d21bc1066387178a63d18d00bc022f85e2ab0cca33b5359771c1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-a, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222) 2024-03-10T03:51:49.014 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 10 03:51:48 smithi006 bash[70839]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-a 2024-03-10T03:51:49.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mon.a.service' 2024-03-10T03:51:49.200 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T03:51:49.200 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-10T03:51:49.201 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-10T03:51:49.201 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mon.c 2024-03-10T03:51:49.336 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:49 smithi006 systemd[1]: Stopping Ceph mon.c for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:51:49.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:49 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-c[31836]: 2024-03-10T03:51:49.541+0000 7ffb0b883700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T03:51:49.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:49 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-c[31836]: 2024-03-10T03:51:49.541+0000 7ffb0b883700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-10T03:51:49.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:49 smithi006 podman[70961]: 2024-03-10 03:51:49.593223589 +0000 UTC m=+0.201872491 container died 2f0215995846b6badac8182726be6361d276aabc35f896fb8cc0f97a90578803 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-c, GIT_CLEAN=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240222) 2024-03-10T03:51:49.837 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:49 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:51:49] ENGINE Bus STOPPING 2024-03-10T03:51:49.837 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:49 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:51:49] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-10T03:51:49.837 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:49 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:51:49] ENGINE Bus STOPPED 2024-03-10T03:51:49.837 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:49 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:51:49] ENGINE Bus STARTING 2024-03-10T03:51:49.837 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:49 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:51:49] ENGINE Serving on http://:::9283 2024-03-10T03:51:49.838 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:49 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: [10/Mar/2024:03:51:49] ENGINE Bus STARTED 2024-03-10T03:51:49.995 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:51:49 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a[47087]: debug there is no tcmu-runner data available 2024-03-10T03:51:50.235 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:50 smithi006 podman[70961]: 2024-03-10 03:51:50.034204514 +0000 UTC m=+0.642853427 container cleanup 2f0215995846b6badac8182726be6361d276aabc35f896fb8cc0f97a90578803 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-10T03:51:50.235 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 10 03:51:50 smithi006 bash[70961]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-c 2024-03-10T03:51:50.452 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mon.c.service' 2024-03-10T03:51:50.489 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T03:51:50.489 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-10T03:51:50.489 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-10T03:51:50.489 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mon.b 2024-03-10T03:51:50.840 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:50 smithi071 systemd[1]: Stopping Ceph mon.b for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:51:51.465 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:51:51 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[70283]: ts=2024-03-10T03:51:51.100Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.00262666s 2024-03-10T03:51:52.090 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:51 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-b[27177]: 2024-03-10T03:51:51.790+0000 7f9339958700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T03:51:52.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:51 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-b[27177]: 2024-03-10T03:51:51.790+0000 7f9339958700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-10T03:51:52.091 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:51 smithi071 podman[60772]: 2024-03-10 03:51:51.88327502 +0000 UTC m=+0.207757790 container died e6f6d97fafa3d71628a42c4a3e70c0bdbb34c30acc54c464df0282cf620c191d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240222) 2024-03-10T03:51:52.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:52 smithi071 podman[60772]: 2024-03-10 03:51:52.255805915 +0000 UTC m=+0.580288684 container cleanup e6f6d97fafa3d71628a42c4a3e70c0bdbb34c30acc54c464df0282cf620c191d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-b, org.label-schema.schema-version=1.0, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5) 2024-03-10T03:51:52.591 INFO:journalctl@ceph.mon.b.smithi071.stdout:Mar 10 03:51:52 smithi071 bash[60772]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mon-b 2024-03-10T03:51:52.825 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mon.b.service' 2024-03-10T03:51:52.883 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T03:51:52.883 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-10T03:51:52.883 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-10T03:51:52.883 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mgr.y 2024-03-10T03:51:53.086 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:52 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: ::ffff:172.21.15.71 - - [10/Mar/2024:03:51:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-10T03:51:53.086 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:52 smithi006 systemd[1]: Stopping Ceph mgr.y for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:51:54.336 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:54 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:51:54.030+0000 7faa9e3ce700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T03:51:54.336 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:54 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y[25487]: 2024-03-10T03:51:54.030+0000 7faa9e3ce700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-10T03:51:54.336 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:54 smithi006 podman[71602]: 2024-03-10 03:51:54.062220162 +0000 UTC m=+0.147453639 container died d42af8aedd6c72411f53bc60c32bd9f1833455fbde23773636b5c3403be7ef30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-10T03:51:54.750 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:54 smithi006 podman[71602]: 2024-03-10 03:51:54.53210443 +0000 UTC m=+0.617337952 container cleanup d42af8aedd6c72411f53bc60c32bd9f1833455fbde23773636b5c3403be7ef30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:831bc188fa69f143bbba69ac76119e784500e8e3, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.schema-version=1.0) 2024-03-10T03:51:54.750 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 10 03:51:54 smithi006 bash[71602]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-y 2024-03-10T03:51:54.886 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mgr.y.service' 2024-03-10T03:51:54.924 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T03:51:54.925 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-10T03:51:54.925 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-10T03:51:54.925 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mgr.x 2024-03-10T03:51:55.225 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:51:54 smithi071 systemd[1]: Stopping Ceph mgr.x for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:51:55.840 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:51:55 smithi071 podman[60965]: 2024-03-10 03:51:55.579832661 +0000 UTC m=+0.340175633 container died 14276e06b3cbc857d23c9f7756ac06b8ccfbc13109ed149e17f6fccd7daa5a36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222) 2024-03-10T03:51:56.341 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:51:55 smithi071 podman[60965]: 2024-03-10 03:51:55.965672925 +0000 UTC m=+0.726015885 container cleanup 14276e06b3cbc857d23c9f7756ac06b8ccfbc13109ed149e17f6fccd7daa5a36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-10T03:51:56.341 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:51:55 smithi071 bash[60965]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x 2024-03-10T03:51:56.341 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:51:56 smithi071 podman[60986]: 2024-03-10 03:51:56.125479609 +0000 UTC m=+0.545226161 container remove 14276e06b3cbc857d23c9f7756ac06b8ccfbc13109ed149e17f6fccd7daa5a36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-mgr-x, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-10T03:51:56.341 INFO:journalctl@ceph.mgr.x.smithi071.stdout:Mar 10 03:51:56 smithi071 systemd[1]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-10T03:51:56.867 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@mgr.x.service' 2024-03-10T03:51:56.908 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T03:51:56.908 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-10T03:51:56.908 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-10T03:51:56.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.0 2024-03-10T03:51:57.298 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 10 03:51:56 smithi006 systemd[1]: Stopping Ceph osd.0 for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:51:58.336 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 10 03:51:57 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-0[36364]: 2024-03-10T03:51:57.920+0000 7f1454237700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T03:51:58.336 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 10 03:51:57 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-0[36364]: 2024-03-10T03:51:57.920+0000 7f1454237700 -1 osd.0 73 *** Got signal Terminated *** 2024-03-10T03:51:58.336 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 10 03:51:57 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-0[36364]: 2024-03-10T03:51:57.920+0000 7f1454237700 -1 osd.0 73 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-10T03:52:03.586 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 10 03:52:03 smithi006 podman[71798]: 2024-03-10 03:52:03.297538356 +0000 UTC m=+5.467726401 container died 932db93cb2f93840b0d9df851f4c3b2faa876e43434fbe0f1e0a778dd5bd7443 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-10T03:52:03.992 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 10 03:52:03 smithi006 podman[71798]: 2024-03-10 03:52:03.766195763 +0000 UTC m=+5.936383843 container cleanup 932db93cb2f93840b0d9df851f4c3b2faa876e43434fbe0f1e0a778dd5bd7443 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True) 2024-03-10T03:52:03.992 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 10 03:52:03 smithi006 bash[71798]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-0 2024-03-10T03:52:04.314 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 10 03:52:03 smithi006 podman[71896]: 2024-03-10 03:52:03.99246599 +0000 UTC m=+0.697350169 container remove 932db93cb2f93840b0d9df851f4c3b2faa876e43434fbe0f1e0a778dd5bd7443 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-0, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.schema-version=1.0) 2024-03-10T03:52:04.586 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 10 03:52:04 smithi006 podman[71990]: 2024-03-10 03:52:04.216444749 +0000 UTC m=+0.018913726 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada 2024-03-10T03:52:04.586 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 10 03:52:04 smithi006 podman[71990]: 2024-03-10 03:52:04.343675744 +0000 UTC m=+0.146144702 container create b9c8a7dd07dfa30627d5afac104b2868a68c313e94998d1fd451964613bb2f20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-10T03:52:04.587 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 10 03:52:04 smithi006 podman[71990]: 2024-03-10 03:52:04.55406649 +0000 UTC m=+0.356535455 container init b9c8a7dd07dfa30627d5afac104b2868a68c313e94998d1fd451964613bb2f20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-10T03:52:04.587 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 10 03:52:04 smithi006 podman[71990]: 2024-03-10 03:52:04.5575596 +0000 UTC m=+0.360028559 container start b9c8a7dd07dfa30627d5afac104b2868a68c313e94998d1fd451964613bb2f20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-10T03:52:05.086 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 10 03:52:04 smithi006 podman[71990]: 2024-03-10 03:52:04.637282134 +0000 UTC m=+0.439751094 container attach b9c8a7dd07dfa30627d5afac104b2868a68c313e94998d1fd451964613bb2f20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-10T03:52:05.086 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 10 03:52:04 smithi006 podman[71990]: 2024-03-10 03:52:04.853805112 +0000 UTC m=+0.656274082 container died b9c8a7dd07dfa30627d5afac104b2868a68c313e94998d1fd451964613bb2f20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240222, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18) 2024-03-10T03:52:05.525 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.0.service' 2024-03-10T03:52:05.574 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T03:52:05.574 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-10T03:52:05.575 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-10T03:52:05.575 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.1 2024-03-10T03:52:05.761 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:52:05 smithi006 systemd[1]: Stopping Ceph osd.1 for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:52:06.086 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:52:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-1[39809]: 2024-03-10T03:52:05.918+0000 7f11c4670700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T03:52:06.087 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:52:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-1[39809]: 2024-03-10T03:52:05.918+0000 7f11c4670700 -1 osd.1 73 *** Got signal Terminated *** 2024-03-10T03:52:06.087 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:52:05 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-1[39809]: 2024-03-10T03:52:05.918+0000 7f11c4670700 -1 osd.1 73 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-10T03:52:11.620 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:52:11 smithi006 podman[72096]: 2024-03-10 03:52:11.353866745 +0000 UTC m=+5.548863766 container died 7ab673529cd236ec4f348c58810c096db3c4330215821dffbe6694fea6c2ba11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-1, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-10T03:52:12.086 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:52:11 smithi006 podman[72096]: 2024-03-10 03:52:11.819691895 +0000 UTC m=+6.014688949 container cleanup 7ab673529cd236ec4f348c58810c096db3c4330215821dffbe6694fea6c2ba11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, maintainer=Guillaume Abrioux ) 2024-03-10T03:52:12.086 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:52:11 smithi006 bash[72096]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-1 2024-03-10T03:52:13.183 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:52:12 smithi006 podman[72460]: 2024-03-10 03:52:12.840947812 +0000 UTC m=+1.480023366 container remove 7ab673529cd236ec4f348c58810c096db3c4330215821dffbe6694fea6c2ba11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-1, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18) 2024-03-10T03:52:13.184 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:52:12 smithi006 bash[72557]: Error: no container with name or ID "ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-1" found: no such container 2024-03-10T03:52:14.047 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:52:13 smithi006 podman[72668]: 2024-03-10 03:52:13.648630231 +0000 UTC m=+0.648523569 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada 2024-03-10T03:52:14.684 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:52:14 smithi006 podman[72668]: 2024-03-10 03:52:14.415608785 +0000 UTC m=+1.415502178 container create 0c302bd73285dcc3690056c186c42a78a7c631916796950da5aea0d289e2e5a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-1-deactivate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) 2024-03-10T03:52:16.143 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:52:15 smithi006 podman[72668]: 2024-03-10 03:52:15.984396257 +0000 UTC m=+2.984289595 container init 0c302bd73285dcc3690056c186c42a78a7c631916796950da5aea0d289e2e5a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-10T03:52:16.143 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:52:15 smithi006 podman[72668]: 2024-03-10 03:52:15.987786124 +0000 UTC m=+2.987679461 container start 0c302bd73285dcc3690056c186c42a78a7c631916796950da5aea0d289e2e5a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-10T03:52:16.586 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:52:16 smithi006 podman[72668]: 2024-03-10 03:52:16.2517465 +0000 UTC m=+3.251639838 container attach 0c302bd73285dcc3690056c186c42a78a7c631916796950da5aea0d289e2e5a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-1-deactivate, ceph=True, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-10T03:52:16.587 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:52:16 smithi006 podman[72668]: 2024-03-10 03:52:16.288863823 +0000 UTC m=+3.288757160 container died 0c302bd73285dcc3690056c186c42a78a7c631916796950da5aea0d289e2e5a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-10T03:52:17.036 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.1.service' 2024-03-10T03:52:17.050 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:52:17 smithi006 podman[72906]: 2024-03-10 03:52:17.014559066 +0000 UTC m=+0.716790396 container remove 0c302bd73285dcc3690056c186c42a78a7c631916796950da5aea0d289e2e5a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-10T03:52:17.050 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:52:17 smithi006 systemd[1]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-10T03:52:17.050 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:52:17 smithi006 systemd[1]: Stopped Ceph osd.1 for 414ba1e0-de90-11ee-95c7-87774f69a715. 2024-03-10T03:52:17.050 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 10 03:52:17 smithi006 systemd[1]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.1.service: Consumed 4.308s CPU time. 2024-03-10T03:52:17.078 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T03:52:17.079 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-10T03:52:17.079 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-10T03:52:17.079 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.2 2024-03-10T03:52:17.311 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 10 03:52:17 smithi006 systemd[1]: Stopping Ceph osd.2 for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:52:18.336 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 10 03:52:17 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-2[43581]: 2024-03-10T03:52:17.938+0000 7fa1eaef2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T03:52:18.336 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 10 03:52:17 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-2[43581]: 2024-03-10T03:52:17.938+0000 7fa1eaef2700 -1 osd.2 73 *** Got signal Terminated *** 2024-03-10T03:52:18.336 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 10 03:52:17 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-2[43581]: 2024-03-10T03:52:17.938+0000 7fa1eaef2700 -1 osd.2 73 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-10T03:52:23.541 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 10 03:52:23 smithi006 podman[73065]: 2024-03-10 03:52:23.282985751 +0000 UTC m=+5.448182484 container died 1b97253c2292122b5d5abca2c935c346f2a5659f10a859040c017d96e88714d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-10T03:52:23.837 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 10 03:52:23 smithi006 podman[73065]: 2024-03-10 03:52:23.714974703 +0000 UTC m=+5.880171435 container cleanup 1b97253c2292122b5d5abca2c935c346f2a5659f10a859040c017d96e88714d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-10T03:52:23.837 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 10 03:52:23 smithi006 bash[73065]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-2 2024-03-10T03:52:24.317 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 10 03:52:23 smithi006 podman[73083]: 2024-03-10 03:52:23.924785612 +0000 UTC m=+0.637342747 container remove 1b97253c2292122b5d5abca2c935c346f2a5659f10a859040c017d96e88714d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-2, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-10T03:52:24.586 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 10 03:52:24 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-3[47139]: 2024-03-10T03:52:24.374+0000 7fb690483700 -1 osd.3 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:51:59.187805+0000 front 2024-03-10T03:51:59.187743+0000 (oldest deadline 2024-03-10T03:52:23.887785+0000) 2024-03-10T03:52:24.586 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 10 03:52:24 smithi006 podman[73172]: 2024-03-10 03:52:24.317486532 +0000 UTC m=+0.099580456 container create 3589d80a3c04b1034a2df7e730abc08c1a861b4dc9081ffbea4cd6a9725e8a9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-10T03:52:24.587 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 10 03:52:24 smithi006 podman[73172]: 2024-03-10 03:52:24.235830074 +0000 UTC m=+0.017923971 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada 2024-03-10T03:52:24.587 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 10 03:52:24 smithi006 podman[73172]: 2024-03-10 03:52:24.552952813 +0000 UTC m=+0.335046696 container init 3589d80a3c04b1034a2df7e730abc08c1a861b4dc9081ffbea4cd6a9725e8a9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-10T03:52:24.587 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 10 03:52:24 smithi006 podman[73172]: 2024-03-10 03:52:24.555983768 +0000 UTC m=+0.338077653 container start 3589d80a3c04b1034a2df7e730abc08c1a861b4dc9081ffbea4cd6a9725e8a9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-2-deactivate, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-10T03:52:24.591 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:24 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:24.120+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:25.086 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 10 03:52:24 smithi006 podman[73172]: 2024-03-10 03:52:24.644746931 +0000 UTC m=+0.426840834 container attach 3589d80a3c04b1034a2df7e730abc08c1a861b4dc9081ffbea4cd6a9725e8a9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, ceph=True, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-10T03:52:25.086 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 10 03:52:24 smithi006 podman[73172]: 2024-03-10 03:52:24.858114853 +0000 UTC m=+0.640208736 container died 3589d80a3c04b1034a2df7e730abc08c1a861b4dc9081ffbea4cd6a9725e8a9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-10T03:52:25.090 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:24 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:24.795+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:25.523 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 10 03:52:25 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-3[47139]: 2024-03-10T03:52:25.355+0000 7fb690483700 -1 osd.3 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:51:59.187805+0000 front 2024-03-10T03:51:59.187743+0000 (oldest deadline 2024-03-10T03:52:23.887785+0000) 2024-03-10T03:52:25.552 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.2.service' 2024-03-10T03:52:25.590 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:25 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:25.136+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:25.596 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T03:52:25.597 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-10T03:52:25.597 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-10T03:52:25.597 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.3 2024-03-10T03:52:25.836 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 10 03:52:25 smithi006 systemd[1]: Stopping Ceph osd.3 for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:52:26.090 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:25 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:25.799+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:26.336 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 10 03:52:25 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-3[47139]: 2024-03-10T03:52:25.892+0000 7fb693e8f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T03:52:26.337 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 10 03:52:25 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-3[47139]: 2024-03-10T03:52:25.892+0000 7fb693e8f700 -1 osd.3 73 *** Got signal Terminated *** 2024-03-10T03:52:26.337 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 10 03:52:25 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-3[47139]: 2024-03-10T03:52:25.892+0000 7fb693e8f700 -1 osd.3 73 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-10T03:52:26.590 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:26 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:26.127+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:26.836 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 10 03:52:26 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-3[47139]: 2024-03-10T03:52:26.380+0000 7fb690483700 -1 osd.3 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:51:59.187805+0000 front 2024-03-10T03:51:59.187743+0000 (oldest deadline 2024-03-10T03:52:23.887785+0000) 2024-03-10T03:52:27.091 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:26 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:26.758+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:27.468 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:27 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:27.100+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:27.469 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:27 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:27.119+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:27.836 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 10 03:52:27 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-3[47139]: 2024-03-10T03:52:27.423+0000 7fb690483700 -1 osd.3 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:51:59.187805+0000 front 2024-03-10T03:51:59.187743+0000 (oldest deadline 2024-03-10T03:52:23.887785+0000) 2024-03-10T03:52:28.091 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:27 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:27.770+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:28.591 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:28 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:28.103+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:28.591 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:28 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:28.138+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:28.836 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 10 03:52:28 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-3[47139]: 2024-03-10T03:52:28.459+0000 7fb690483700 -1 osd.3 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:51:59.187805+0000 front 2024-03-10T03:51:59.187743+0000 (oldest deadline 2024-03-10T03:52:23.887785+0000) 2024-03-10T03:52:29.046 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:28 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:28.782+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:29.341 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:29 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:29.065+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:29.341 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:29 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:52:29.045+0000 7fc30ccea700 -1 osd.4 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.419310+0000 front 2024-03-10T03:52:02.419220+0000 (oldest deadline 2024-03-10T03:52:28.319199+0000) 2024-03-10T03:52:29.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:29 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:29.156+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:29.837 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 10 03:52:29 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-3[47139]: 2024-03-10T03:52:29.444+0000 7fb690483700 -1 osd.3 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:51:59.187805+0000 front 2024-03-10T03:51:59.187743+0000 (oldest deadline 2024-03-10T03:52:23.887785+0000) 2024-03-10T03:52:30.091 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:30 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:52:30.041+0000 7fc30ccea700 -1 osd.4 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.419310+0000 front 2024-03-10T03:52:02.419220+0000 (oldest deadline 2024-03-10T03:52:28.319199+0000) 2024-03-10T03:52:30.091 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:29 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:29.832+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:30.091 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:30 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:30.079+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:30.590 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:30 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:30.162+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:30.836 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 10 03:52:30 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-3[47139]: 2024-03-10T03:52:30.452+0000 7fb690483700 -1 osd.3 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:51:59.187805+0000 front 2024-03-10T03:51:59.187743+0000 (oldest deadline 2024-03-10T03:52:23.887785+0000) 2024-03-10T03:52:31.132 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:31 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:52:31.000+0000 7fc30ccea700 -1 osd.4 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.419310+0000 front 2024-03-10T03:52:02.419220+0000 (oldest deadline 2024-03-10T03:52:28.319199+0000) 2024-03-10T03:52:31.132 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:30 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:30.860+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:31.132 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:31 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:31.054+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:31.593 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:31 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:31.131+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:31.685 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 10 03:52:31 smithi006 podman[73277]: 2024-03-10 03:52:31.402737888 +0000 UTC m=+5.616618397 container died fc197ceb381f53bf958a8188664efde302c9f477f3bf2fe41a75fdb19f3d49a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-3, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-10T03:52:31.981 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 10 03:52:31 smithi006 podman[73277]: 2024-03-10 03:52:31.876546386 +0000 UTC m=+6.090426930 container cleanup fc197ceb381f53bf958a8188664efde302c9f477f3bf2fe41a75fdb19f3d49a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-3, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-10T03:52:31.981 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 10 03:52:31 smithi006 bash[73277]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-3 2024-03-10T03:52:32.341 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:32 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:52:32.001+0000 7fc30ccea700 -1 osd.4 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.419310+0000 front 2024-03-10T03:52:02.419220+0000 (oldest deadline 2024-03-10T03:52:28.319199+0000) 2024-03-10T03:52:32.341 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:32 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:32.083+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:32.341 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:31 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:31.854+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:32.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:32 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:32.096+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:33.090 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:33 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:52:33.048+0000 7fc30ccea700 -1 osd.4 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.419310+0000 front 2024-03-10T03:52:02.419220+0000 (oldest deadline 2024-03-10T03:52:28.319199+0000) 2024-03-10T03:52:33.091 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:32 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:32.829+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:33.091 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:32 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:32.829+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:08.565485+0000 front 2024-03-10T03:52:08.565582+0000 (oldest deadline 2024-03-10T03:52:32.665504+0000) 2024-03-10T03:52:33.591 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:33 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:33.098+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:33.591 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:33 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:33.137+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:33.591 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:33 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:33.137+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:33.787 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 10 03:52:33 smithi006 bash[73397]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-3 2024-03-10T03:52:34.043 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 10 03:52:33 smithi006 podman[73295]: 2024-03-10 03:52:33.787350325 +0000 UTC m=+2.375631440 container remove fc197ceb381f53bf958a8188664efde302c9f477f3bf2fe41a75fdb19f3d49a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-3, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-10T03:52:34.085 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:34 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:52:34.007+0000 7fc30ccea700 -1 osd.4 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.419310+0000 front 2024-03-10T03:52:02.419220+0000 (oldest deadline 2024-03-10T03:52:28.319199+0000) 2024-03-10T03:52:34.085 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:34 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:52:34.007+0000 7fc30ccea700 -1 osd.4 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.420332+0000 front 2024-03-10T03:52:09.420225+0000 (oldest deadline 2024-03-10T03:52:33.520089+0000) 2024-03-10T03:52:34.085 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:33 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:33.833+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:34.085 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:33 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:33.833+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:08.565485+0000 front 2024-03-10T03:52:08.565582+0000 (oldest deadline 2024-03-10T03:52:32.665504+0000) 2024-03-10T03:52:34.305 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 10 03:52:34 smithi006 podman[73629]: 2024-03-10 03:52:33.946249777 +0000 UTC m=+0.019722441 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada 2024-03-10T03:52:34.305 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 10 03:52:34 smithi006 podman[73629]: 2024-03-10 03:52:34.046486531 +0000 UTC m=+0.119959216 container create 8317f09e50a9099f04a045940b05eea241a3fe16f11322348b894c8d7d49e0df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, GIT_CLEAN=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-10T03:52:34.341 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:34 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:34.084+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:34.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:34 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:34.119+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:34.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:34 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:34.119+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:34.836 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 10 03:52:34 smithi006 podman[73629]: 2024-03-10 03:52:34.794354304 +0000 UTC m=+0.867826950 container init 8317f09e50a9099f04a045940b05eea241a3fe16f11322348b894c8d7d49e0df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18) 2024-03-10T03:52:34.836 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 10 03:52:34 smithi006 podman[73629]: 2024-03-10 03:52:34.797615629 +0000 UTC m=+0.871088278 container start 8317f09e50a9099f04a045940b05eea241a3fe16f11322348b894c8d7d49e0df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-10T03:52:34.961 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:34 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:34.868+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:34.961 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:34 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:34.868+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:08.565485+0000 front 2024-03-10T03:52:08.565582+0000 (oldest deadline 2024-03-10T03:52:32.665504+0000) 2024-03-10T03:52:35.272 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 10 03:52:34 smithi006 podman[73629]: 2024-03-10 03:52:34.903018602 +0000 UTC m=+0.976491253 container attach 8317f09e50a9099f04a045940b05eea241a3fe16f11322348b894c8d7d49e0df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18) 2024-03-10T03:52:35.272 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 10 03:52:35 smithi006 podman[73629]: 2024-03-10 03:52:35.130636989 +0000 UTC m=+1.204109633 container died 8317f09e50a9099f04a045940b05eea241a3fe16f11322348b894c8d7d49e0df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222) 2024-03-10T03:52:35.341 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:35 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:35.060+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:35.341 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:34 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:52:34.960+0000 7fc30ccea700 -1 osd.4 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.419310+0000 front 2024-03-10T03:52:02.419220+0000 (oldest deadline 2024-03-10T03:52:28.319199+0000) 2024-03-10T03:52:35.341 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:34 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:52:34.960+0000 7fc30ccea700 -1 osd.4 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.420332+0000 front 2024-03-10T03:52:09.420225+0000 (oldest deadline 2024-03-10T03:52:33.520089+0000) 2024-03-10T03:52:35.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:35 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:35.169+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:35.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:35 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:35.169+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:35.695 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.3.service' 2024-03-10T03:52:35.735 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T03:52:35.736 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-10T03:52:35.736 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-10T03:52:35.736 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.4 2024-03-10T03:52:35.994 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:35 smithi071 systemd[1]: Stopping Ceph osd.4 for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:52:35.994 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:35 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:52:35.967+0000 7fc30ccea700 -1 osd.4 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.419310+0000 front 2024-03-10T03:52:02.419220+0000 (oldest deadline 2024-03-10T03:52:28.319199+0000) 2024-03-10T03:52:35.995 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:35 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:52:35.967+0000 7fc30ccea700 -1 osd.4 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.420332+0000 front 2024-03-10T03:52:09.420225+0000 (oldest deadline 2024-03-10T03:52:33.520089+0000) 2024-03-10T03:52:35.995 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:35 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:35.883+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:35.995 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:35 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:35.883+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:08.565485+0000 front 2024-03-10T03:52:08.565582+0000 (oldest deadline 2024-03-10T03:52:32.665504+0000) 2024-03-10T03:52:36.300 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:36 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:36.033+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:36.300 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:36 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:36.033+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.969210+0000 front 2024-03-10T03:52:09.969238+0000 (oldest deadline 2024-03-10T03:52:35.269187+0000) 2024-03-10T03:52:36.300 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:36 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:36.155+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:36.300 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:36 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:36.155+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:36.551 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:36 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:52:36.299+0000 7fc3146fe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T03:52:36.551 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:36 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:52:36.299+0000 7fc3146fe700 -1 osd.4 73 *** Got signal Terminated *** 2024-03-10T03:52:36.551 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:36 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:52:36.299+0000 7fc3146fe700 -1 osd.4 73 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-10T03:52:37.181 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:36 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:36.903+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:37.181 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:36 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:36.903+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:08.565485+0000 front 2024-03-10T03:52:08.565582+0000 (oldest deadline 2024-03-10T03:52:32.665504+0000) 2024-03-10T03:52:37.181 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:37 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:37.045+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:37.181 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:37 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:37.045+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.969210+0000 front 2024-03-10T03:52:09.969238+0000 (oldest deadline 2024-03-10T03:52:35.269187+0000) 2024-03-10T03:52:37.182 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:36 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:52:36.964+0000 7fc30ccea700 -1 osd.4 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.419310+0000 front 2024-03-10T03:52:02.419220+0000 (oldest deadline 2024-03-10T03:52:28.319199+0000) 2024-03-10T03:52:37.182 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:36 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:52:36.964+0000 7fc30ccea700 -1 osd.4 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.420332+0000 front 2024-03-10T03:52:09.420225+0000 (oldest deadline 2024-03-10T03:52:33.520089+0000) 2024-03-10T03:52:37.488 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:37 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:37.180+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:37.488 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:37 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:37.180+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:38.154 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:37 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:52:37.987+0000 7fc30ccea700 -1 osd.4 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.419310+0000 front 2024-03-10T03:52:02.419220+0000 (oldest deadline 2024-03-10T03:52:28.319199+0000) 2024-03-10T03:52:38.154 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:37 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:52:37.987+0000 7fc30ccea700 -1 osd.4 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.420332+0000 front 2024-03-10T03:52:09.420225+0000 (oldest deadline 2024-03-10T03:52:33.520089+0000) 2024-03-10T03:52:38.154 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:38 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:38.008+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:38.154 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:38 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:38.008+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.969210+0000 front 2024-03-10T03:52:09.969238+0000 (oldest deadline 2024-03-10T03:52:35.269187+0000) 2024-03-10T03:52:38.154 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:37 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:37.865+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:38.155 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:37 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:37.865+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:08.565485+0000 front 2024-03-10T03:52:08.565582+0000 (oldest deadline 2024-03-10T03:52:32.665504+0000) 2024-03-10T03:52:38.591 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:38 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:38.152+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:38.591 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:38 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:38.153+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:39.198 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:39 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:39.054+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:39.198 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:39 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:39.054+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.969210+0000 front 2024-03-10T03:52:09.969238+0000 (oldest deadline 2024-03-10T03:52:35.269187+0000) 2024-03-10T03:52:39.199 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:38 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:52:38.940+0000 7fc30ccea700 -1 osd.4 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.419310+0000 front 2024-03-10T03:52:02.419220+0000 (oldest deadline 2024-03-10T03:52:28.319199+0000) 2024-03-10T03:52:39.199 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:38 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:52:38.940+0000 7fc30ccea700 -1 osd.4 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.420332+0000 front 2024-03-10T03:52:09.420225+0000 (oldest deadline 2024-03-10T03:52:33.520089+0000) 2024-03-10T03:52:39.199 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:38 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:38.887+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:39.199 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:38 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:38.887+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:08.565485+0000 front 2024-03-10T03:52:08.565582+0000 (oldest deadline 2024-03-10T03:52:32.665504+0000) 2024-03-10T03:52:39.590 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:39 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:39.197+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:39.591 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:39 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:39.197+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:40.199 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:39 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:52:39.970+0000 7fc30ccea700 -1 osd.4 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.419310+0000 front 2024-03-10T03:52:02.419220+0000 (oldest deadline 2024-03-10T03:52:28.319199+0000) 2024-03-10T03:52:40.199 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:39 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:52:39.970+0000 7fc30ccea700 -1 osd.4 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.420332+0000 front 2024-03-10T03:52:09.420225+0000 (oldest deadline 2024-03-10T03:52:33.520089+0000) 2024-03-10T03:52:40.199 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:40 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:40.046+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:40.199 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:40 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:40.046+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.969210+0000 front 2024-03-10T03:52:09.969238+0000 (oldest deadline 2024-03-10T03:52:35.269187+0000) 2024-03-10T03:52:40.199 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:39 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:39.898+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:40.199 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:39 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:39.898+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:08.565485+0000 front 2024-03-10T03:52:08.565582+0000 (oldest deadline 2024-03-10T03:52:32.665504+0000) 2024-03-10T03:52:40.590 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:40 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:40.198+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:40.591 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:40 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:40.198+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:41.196 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:41 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:41.074+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:41.196 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:41 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:41.074+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.969210+0000 front 2024-03-10T03:52:09.969238+0000 (oldest deadline 2024-03-10T03:52:35.269187+0000) 2024-03-10T03:52:41.196 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:40 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:52:40.990+0000 7fc30ccea700 -1 osd.4 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.419310+0000 front 2024-03-10T03:52:02.419220+0000 (oldest deadline 2024-03-10T03:52:28.319199+0000) 2024-03-10T03:52:41.196 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:40 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4[33665]: 2024-03-10T03:52:40.990+0000 7fc30ccea700 -1 osd.4 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.420332+0000 front 2024-03-10T03:52:09.420225+0000 (oldest deadline 2024-03-10T03:52:33.520089+0000) 2024-03-10T03:52:41.197 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:40 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:40.893+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:41.197 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:40 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:40.893+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:08.565485+0000 front 2024-03-10T03:52:08.565582+0000 (oldest deadline 2024-03-10T03:52:32.665504+0000) 2024-03-10T03:52:41.590 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:41 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:41.195+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:41.591 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:41 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:41.195+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:41.943 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:41 smithi071 podman[62101]: 2024-03-10 03:52:41.63139057 +0000 UTC m=+5.442862907 container died 9ea9b48c43d5b33fad71197ce275dda61c02d494a0f35b7052258d72b07cf9cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-10T03:52:42.245 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:41 smithi071 podman[62101]: 2024-03-10 03:52:41.97852784 +0000 UTC m=+5.790000174 container cleanup 9ea9b48c43d5b33fad71197ce275dda61c02d494a0f35b7052258d72b07cf9cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-10T03:52:42.245 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:41 smithi071 bash[62101]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4 2024-03-10T03:52:42.245 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:42 smithi071 podman[62211]: 2024-03-10 03:52:42.146729838 +0000 UTC m=+0.513386676 container remove 9ea9b48c43d5b33fad71197ce275dda61c02d494a0f35b7052258d72b07cf9cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-10T03:52:42.245 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:41 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:41.941+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:42.246 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:41 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:41.941+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:08.565485+0000 front 2024-03-10T03:52:08.565582+0000 (oldest deadline 2024-03-10T03:52:32.665504+0000) 2024-03-10T03:52:42.246 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:42 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:42.115+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:42.246 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:42 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:42.115+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.969210+0000 front 2024-03-10T03:52:09.969238+0000 (oldest deadline 2024-03-10T03:52:35.269187+0000) 2024-03-10T03:52:42.544 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:42 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:42.243+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:42.544 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:42 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:42.243+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:42.808 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:42 smithi071 podman[62304]: 2024-03-10 03:52:42.447052864 +0000 UTC m=+0.018508256 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada 2024-03-10T03:52:42.808 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:42 smithi071 podman[62304]: 2024-03-10 03:52:42.564853819 +0000 UTC m=+0.136309204 container create 6861d812a936ab47e55998f58178f19de0d4b6f3a7448fcabcd6af28af3e31ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-10T03:52:42.808 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:42 smithi071 podman[62304]: 2024-03-10 03:52:42.808330579 +0000 UTC m=+0.379785972 container init 6861d812a936ab47e55998f58178f19de0d4b6f3a7448fcabcd6af28af3e31ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-10T03:52:43.091 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:42 smithi071 podman[62304]: 2024-03-10 03:52:42.811487583 +0000 UTC m=+0.382942974 container start 6861d812a936ab47e55998f58178f19de0d4b6f3a7448fcabcd6af28af3e31ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-10T03:52:43.091 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:42 smithi071 podman[62304]: 2024-03-10 03:52:42.891409386 +0000 UTC m=+0.462864783 container attach 6861d812a936ab47e55998f58178f19de0d4b6f3a7448fcabcd6af28af3e31ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-10T03:52:43.091 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:42 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:42.949+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:43.091 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:42 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:42.949+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:08.565485+0000 front 2024-03-10T03:52:08.565582+0000 (oldest deadline 2024-03-10T03:52:32.665504+0000) 2024-03-10T03:52:43.394 INFO:journalctl@ceph.osd.4.smithi071.stdout:Mar 10 03:52:43 smithi071 podman[62304]: 2024-03-10 03:52:43.110066736 +0000 UTC m=+0.681522130 container died 6861d812a936ab47e55998f58178f19de0d4b6f3a7448fcabcd6af28af3e31ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-10T03:52:43.394 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:43 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:43.109+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:43.395 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:43 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:43.109+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.969210+0000 front 2024-03-10T03:52:09.969238+0000 (oldest deadline 2024-03-10T03:52:35.269187+0000) 2024-03-10T03:52:43.395 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:43 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:43.221+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:43.395 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:43 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:43.221+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:43.743 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.4.service' 2024-03-10T03:52:43.787 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T03:52:43.787 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-10T03:52:43.787 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-10T03:52:43.788 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.5 2024-03-10T03:52:44.091 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:43 smithi071 systemd[1]: Stopping Ceph osd.5 for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:52:44.091 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:43 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:43.949+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:44.091 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:43 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:43.949+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:08.565485+0000 front 2024-03-10T03:52:08.565582+0000 (oldest deadline 2024-03-10T03:52:32.665504+0000) 2024-03-10T03:52:44.430 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:44.148+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:44.430 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:44.148+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.969210+0000 front 2024-03-10T03:52:09.969238+0000 (oldest deadline 2024-03-10T03:52:35.269187+0000) 2024-03-10T03:52:44.431 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:44.148+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:22.771171+0000 front 2024-03-10T03:52:22.771145+0000 (oldest deadline 2024-03-10T03:52:43.771599+0000) 2024-03-10T03:52:44.431 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:44.113+0000 7ff6fbf6f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T03:52:44.431 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:44.113+0000 7ff6fbf6f700 -1 osd.5 73 *** Got signal Terminated *** 2024-03-10T03:52:44.431 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:44.114+0000 7ff6fbf6f700 -1 osd.5 73 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-10T03:52:44.431 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:44.245+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:44.431 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:44.245+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:44.431 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:44.245+0000 7faf0da5b700 -1 osd.7 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24424.0:155 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73) 2024-03-10T03:52:45.225 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:45 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:45.126+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:45.225 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:45 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:45.126+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.969210+0000 front 2024-03-10T03:52:09.969238+0000 (oldest deadline 2024-03-10T03:52:35.269187+0000) 2024-03-10T03:52:45.226 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:45 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:45.126+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:22.771171+0000 front 2024-03-10T03:52:22.771145+0000 (oldest deadline 2024-03-10T03:52:43.771599+0000) 2024-03-10T03:52:45.226 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:44.950+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:45.226 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:44 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:44.950+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:08.565485+0000 front 2024-03-10T03:52:08.565582+0000 (oldest deadline 2024-03-10T03:52:32.665504+0000) 2024-03-10T03:52:45.590 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:45 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:45.224+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:45.591 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:45 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:45.224+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:45.591 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:45 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:45.224+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:19.783838+0000 front 2024-03-10T03:52:19.783807+0000 (oldest deadline 2024-03-10T03:52:45.083769+0000) 2024-03-10T03:52:45.591 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:45 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:45.224+0000 7faf0da5b700 -1 osd.7 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24424.0:155 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73) 2024-03-10T03:52:46.142 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:45 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:45.924+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:46.142 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:45 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:45.924+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:08.565485+0000 front 2024-03-10T03:52:08.565582+0000 (oldest deadline 2024-03-10T03:52:32.665504+0000) 2024-03-10T03:52:46.575 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:46.141+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:46.575 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:46.141+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.969210+0000 front 2024-03-10T03:52:09.969238+0000 (oldest deadline 2024-03-10T03:52:35.269187+0000) 2024-03-10T03:52:46.575 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:46.141+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:22.771171+0000 front 2024-03-10T03:52:22.771145+0000 (oldest deadline 2024-03-10T03:52:43.771599+0000) 2024-03-10T03:52:46.575 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:46.180+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:46.575 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:46.180+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:46.575 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:46.180+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:19.783838+0000 front 2024-03-10T03:52:19.783807+0000 (oldest deadline 2024-03-10T03:52:45.083769+0000) 2024-03-10T03:52:46.575 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:46.180+0000 7faf0da5b700 -1 osd.7 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24424.0:155 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73) 2024-03-10T03:52:46.576 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:52:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:52:46.238Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.6:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.6:8765: connect: connection refused" 2024-03-10T03:52:46.576 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:52:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:52:46.241Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.6:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.6:8765: connect: connection refused" 2024-03-10T03:52:46.576 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:52:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:52:46.241Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.6:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.6:8765: connect: connection refused" 2024-03-10T03:52:46.576 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:52:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:52:46.243Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.6:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.6:8765: connect: connection refused" 2024-03-10T03:52:46.576 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:52:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:52:46.243Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.6:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.6:8765: connect: connection refused" 2024-03-10T03:52:47.162 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:46.887+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:47.162 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:46 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:46.887+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:08.565485+0000 front 2024-03-10T03:52:08.565582+0000 (oldest deadline 2024-03-10T03:52:32.665504+0000) 2024-03-10T03:52:47.163 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:47 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:47.092+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:47.163 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:47 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:47.092+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.969210+0000 front 2024-03-10T03:52:09.969238+0000 (oldest deadline 2024-03-10T03:52:35.269187+0000) 2024-03-10T03:52:47.163 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:47 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:47.092+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:22.771171+0000 front 2024-03-10T03:52:22.771145+0000 (oldest deadline 2024-03-10T03:52:43.771599+0000) 2024-03-10T03:52:47.438 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:47 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:47.158+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:47.438 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:47 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:47.158+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:47.438 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:47 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:47.158+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:19.783838+0000 front 2024-03-10T03:52:19.783807+0000 (oldest deadline 2024-03-10T03:52:45.083769+0000) 2024-03-10T03:52:47.439 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:47 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:47.158+0000 7faf0da5b700 -1 osd.7 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24424.0:155 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73) 2024-03-10T03:52:48.139 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:47 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:47.882+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:48.140 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:47 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:47.882+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:08.565485+0000 front 2024-03-10T03:52:08.565582+0000 (oldest deadline 2024-03-10T03:52:32.665504+0000) 2024-03-10T03:52:48.140 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:47 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:47.882+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:21.867837+0000 front 2024-03-10T03:52:21.867899+0000 (oldest deadline 2024-03-10T03:52:47.767838+0000) 2024-03-10T03:52:48.590 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:48 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:48.139+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:48.591 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:48 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:48.139+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.969210+0000 front 2024-03-10T03:52:09.969238+0000 (oldest deadline 2024-03-10T03:52:35.269187+0000) 2024-03-10T03:52:48.591 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:48 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:48.139+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:22.771171+0000 front 2024-03-10T03:52:22.771145+0000 (oldest deadline 2024-03-10T03:52:43.771599+0000) 2024-03-10T03:52:48.591 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:48 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:48.166+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:48.591 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:48 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:48.166+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:48.591 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:48 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:48.166+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:19.783838+0000 front 2024-03-10T03:52:19.783807+0000 (oldest deadline 2024-03-10T03:52:45.083769+0000) 2024-03-10T03:52:48.591 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:48 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:48.166+0000 7faf0da5b700 -1 osd.7 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24424.0:155 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73) 2024-03-10T03:52:49.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:49 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:49.140+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:49.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:49 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:49.140+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:49.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:49 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:49.140+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:19.783838+0000 front 2024-03-10T03:52:19.783807+0000 (oldest deadline 2024-03-10T03:52:45.083769+0000) 2024-03-10T03:52:49.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:49 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:49.141+0000 7faf0da5b700 -1 osd.7 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24424.0:155 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73) 2024-03-10T03:52:49.341 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:48 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:48.929+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.264409+0000 front 2024-03-10T03:52:02.264238+0000 (oldest deadline 2024-03-10T03:52:23.964204+0000) 2024-03-10T03:52:49.341 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:48 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:48.929+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:08.565485+0000 front 2024-03-10T03:52:08.565582+0000 (oldest deadline 2024-03-10T03:52:32.665504+0000) 2024-03-10T03:52:49.341 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:48 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5[37046]: 2024-03-10T03:52:48.929+0000 7ff6f455b700 -1 osd.5 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:21.867837+0000 front 2024-03-10T03:52:21.867899+0000 (oldest deadline 2024-03-10T03:52:47.767838+0000) 2024-03-10T03:52:49.342 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:49 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:49.116+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:49.342 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:49 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:49.116+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.969210+0000 front 2024-03-10T03:52:09.969238+0000 (oldest deadline 2024-03-10T03:52:35.269187+0000) 2024-03-10T03:52:49.342 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:49 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:49.116+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:22.771171+0000 front 2024-03-10T03:52:22.771145+0000 (oldest deadline 2024-03-10T03:52:43.771599+0000) 2024-03-10T03:52:49.872 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:49 smithi071 podman[62409]: 2024-03-10 03:52:49.60113488 +0000 UTC m=+5.589352754 container died 979e663e031c2e6c2123439657ad1e80107a629c26854012bf4f3ef8da4aa1b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-10T03:52:50.136 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:50 smithi071 podman[62409]: 2024-03-10 03:52:50.038563526 +0000 UTC m=+6.026781416 container cleanup 979e663e031c2e6c2123439657ad1e80107a629c26854012bf4f3ef8da4aa1b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-10T03:52:50.136 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:50 smithi071 bash[62409]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5 2024-03-10T03:52:50.590 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:50 smithi071 podman[63092]: 2024-03-10 03:52:50.323539027 +0000 UTC m=+0.720368350 container remove 979e663e031c2e6c2123439657ad1e80107a629c26854012bf4f3ef8da4aa1b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18) 2024-03-10T03:52:50.591 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:50 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:50.135+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:50.591 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:50 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:50.135+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.969210+0000 front 2024-03-10T03:52:09.969238+0000 (oldest deadline 2024-03-10T03:52:35.269187+0000) 2024-03-10T03:52:50.591 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:50 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:50.135+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:22.771171+0000 front 2024-03-10T03:52:22.771145+0000 (oldest deadline 2024-03-10T03:52:43.771599+0000) 2024-03-10T03:52:50.591 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:50 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:50.145+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:50.591 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:50 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:50.145+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:50.591 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:50 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:50.145+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:19.783838+0000 front 2024-03-10T03:52:19.783807+0000 (oldest deadline 2024-03-10T03:52:45.083769+0000) 2024-03-10T03:52:50.591 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:50 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:50.145+0000 7faf0da5b700 -1 osd.7 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24424.0:155 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73) 2024-03-10T03:52:51.085 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:50 smithi071 podman[63186]: 2024-03-10 03:52:50.691846407 +0000 UTC m=+0.017947845 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada 2024-03-10T03:52:51.085 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:50 smithi071 podman[63186]: 2024-03-10 03:52:50.799216598 +0000 UTC m=+0.125318022 container create e29c974b6e891dd99bf416354e1f6bb108fafe0564d15e3466ee1a93c0ec25b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-10T03:52:51.085 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:51 smithi071 podman[63186]: 2024-03-10 03:52:51.001516524 +0000 UTC m=+0.327617963 container init e29c974b6e891dd99bf416354e1f6bb108fafe0564d15e3466ee1a93c0ec25b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5-deactivate, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-10T03:52:51.085 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:51 smithi071 podman[63186]: 2024-03-10 03:52:51.004669434 +0000 UTC m=+0.330770864 container start e29c974b6e891dd99bf416354e1f6bb108fafe0564d15e3466ee1a93c0ec25b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-10T03:52:51.085 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:51 smithi071 podman[63186]: 2024-03-10 03:52:51.084697146 +0000 UTC m=+0.410798588 container attach e29c974b6e891dd99bf416354e1f6bb108fafe0564d15e3466ee1a93c0ec25b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240222, ceph=True) 2024-03-10T03:52:51.341 INFO:journalctl@ceph.osd.5.smithi071.stdout:Mar 10 03:52:51 smithi071 podman[63186]: 2024-03-10 03:52:51.309016185 +0000 UTC m=+0.635117614 container died e29c974b6e891dd99bf416354e1f6bb108fafe0564d15e3466ee1a93c0ec25b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-10T03:52:51.341 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:51 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:51.134+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:51.341 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:51 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:51.134+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.969210+0000 front 2024-03-10T03:52:09.969238+0000 (oldest deadline 2024-03-10T03:52:35.269187+0000) 2024-03-10T03:52:51.341 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:51 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:51.134+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:22.771171+0000 front 2024-03-10T03:52:22.771145+0000 (oldest deadline 2024-03-10T03:52:43.771599+0000) 2024-03-10T03:52:51.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:51 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:51.098+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:51.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:51 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:51.098+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:51.342 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:51 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:51.098+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:19.783838+0000 front 2024-03-10T03:52:19.783807+0000 (oldest deadline 2024-03-10T03:52:45.083769+0000) 2024-03-10T03:52:51.342 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:51 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:51.099+0000 7faf0da5b700 -1 osd.7 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24424.0:155 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73) 2024-03-10T03:52:51.827 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.5.service' 2024-03-10T03:52:51.868 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T03:52:51.869 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-10T03:52:51.869 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-10T03:52:51.869 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.6 2024-03-10T03:52:52.064 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:51 smithi071 systemd[1]: Stopping Ceph osd.6 for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:52:52.340 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:52 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:52.062+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:52.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:52 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:52.062+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:52.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:52 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:52.062+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:19.783838+0000 front 2024-03-10T03:52:19.783807+0000 (oldest deadline 2024-03-10T03:52:45.083769+0000) 2024-03-10T03:52:52.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:52 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:52.062+0000 7faf0da5b700 -1 osd.7 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24424.0:155 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73) 2024-03-10T03:52:52.341 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:52 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:52.165+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:52.341 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:52 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:52.165+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.969210+0000 front 2024-03-10T03:52:09.969238+0000 (oldest deadline 2024-03-10T03:52:35.269187+0000) 2024-03-10T03:52:52.341 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:52 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:52.165+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:22.771171+0000 front 2024-03-10T03:52:22.771145+0000 (oldest deadline 2024-03-10T03:52:43.771599+0000) 2024-03-10T03:52:52.341 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:52 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:52.207+0000 7fdc01394700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T03:52:52.341 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:52 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:52.207+0000 7fdc01394700 -1 osd.6 73 *** Got signal Terminated *** 2024-03-10T03:52:52.341 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:52 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:52.207+0000 7fdc01394700 -1 osd.6 73 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-10T03:52:53.341 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:53 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:53.183+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:53.341 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:53 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:53.183+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.969210+0000 front 2024-03-10T03:52:09.969238+0000 (oldest deadline 2024-03-10T03:52:35.269187+0000) 2024-03-10T03:52:53.341 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:53 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:53.183+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:22.771171+0000 front 2024-03-10T03:52:22.771145+0000 (oldest deadline 2024-03-10T03:52:43.771599+0000) 2024-03-10T03:52:53.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:53 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:53.072+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:53.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:53 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:53.072+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:53.342 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:53 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:53.072+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:19.783838+0000 front 2024-03-10T03:52:19.783807+0000 (oldest deadline 2024-03-10T03:52:45.083769+0000) 2024-03-10T03:52:53.342 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:53 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:53.072+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6830 osd.3 since back 2024-03-10T03:52:30.384722+0000 front 2024-03-10T03:52:30.384735+0000 (oldest deadline 2024-03-10T03:52:52.084522+0000) 2024-03-10T03:52:53.342 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:53 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:53.072+0000 7faf0da5b700 -1 osd.7 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24424.0:155 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73) 2024-03-10T03:52:54.341 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:54 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:54.144+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:54.341 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:54 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:54.144+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.969210+0000 front 2024-03-10T03:52:09.969238+0000 (oldest deadline 2024-03-10T03:52:35.269187+0000) 2024-03-10T03:52:54.341 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:54 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:54.144+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:22.771171+0000 front 2024-03-10T03:52:22.771145+0000 (oldest deadline 2024-03-10T03:52:43.771599+0000) 2024-03-10T03:52:54.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:54 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:54.024+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:54.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:54 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:54.024+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:54.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:54 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:54.024+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:19.783838+0000 front 2024-03-10T03:52:19.783807+0000 (oldest deadline 2024-03-10T03:52:45.083769+0000) 2024-03-10T03:52:54.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:54 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:54.024+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6830 osd.3 since back 2024-03-10T03:52:30.384722+0000 front 2024-03-10T03:52:30.384735+0000 (oldest deadline 2024-03-10T03:52:52.084522+0000) 2024-03-10T03:52:54.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:54 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:54.024+0000 7faf0da5b700 -1 osd.7 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24424.0:155 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73) 2024-03-10T03:52:55.341 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:55 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:55.099+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:55.341 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:55 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:55.099+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.969210+0000 front 2024-03-10T03:52:09.969238+0000 (oldest deadline 2024-03-10T03:52:35.269187+0000) 2024-03-10T03:52:55.341 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:55 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:55.099+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:22.771171+0000 front 2024-03-10T03:52:22.771145+0000 (oldest deadline 2024-03-10T03:52:43.771599+0000) 2024-03-10T03:52:55.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:55 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:55.034+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:55.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:55 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:55.034+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:55.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:55 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:55.034+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:19.783838+0000 front 2024-03-10T03:52:19.783807+0000 (oldest deadline 2024-03-10T03:52:45.083769+0000) 2024-03-10T03:52:55.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:55 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:55.034+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6830 osd.3 since back 2024-03-10T03:52:30.384722+0000 front 2024-03-10T03:52:30.384735+0000 (oldest deadline 2024-03-10T03:52:52.084522+0000) 2024-03-10T03:52:55.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:55 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:55.034+0000 7faf0da5b700 -1 osd.7 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24424.0:155 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73) 2024-03-10T03:52:56.054 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:56 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:56.053+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:56.054 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:56 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:56.053+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.969210+0000 front 2024-03-10T03:52:09.969238+0000 (oldest deadline 2024-03-10T03:52:35.269187+0000) 2024-03-10T03:52:56.054 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:56 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:56.053+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:22.771171+0000 front 2024-03-10T03:52:22.771145+0000 (oldest deadline 2024-03-10T03:52:43.771599+0000) 2024-03-10T03:52:56.055 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:55 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:55.998+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:56.055 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:55 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:55.998+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:56.055 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:55 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:55.998+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:19.783838+0000 front 2024-03-10T03:52:19.783807+0000 (oldest deadline 2024-03-10T03:52:45.083769+0000) 2024-03-10T03:52:56.055 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:55 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:55.998+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6830 osd.3 since back 2024-03-10T03:52:30.384722+0000 front 2024-03-10T03:52:30.384735+0000 (oldest deadline 2024-03-10T03:52:52.084522+0000) 2024-03-10T03:52:56.055 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:56 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:55.999+0000 7faf0da5b700 -1 osd.7 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24424.0:155 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73) 2024-03-10T03:52:57.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:56 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:56.990+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:57.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:56 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:56.990+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:57.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:56 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:56.990+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:19.783838+0000 front 2024-03-10T03:52:19.783807+0000 (oldest deadline 2024-03-10T03:52:45.083769+0000) 2024-03-10T03:52:57.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:56 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:56.990+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6830 osd.3 since back 2024-03-10T03:52:30.384722+0000 front 2024-03-10T03:52:30.384735+0000 (oldest deadline 2024-03-10T03:52:52.084522+0000) 2024-03-10T03:52:57.342 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:56 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:56.990+0000 7faf0da5b700 -1 osd.7 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24424.0:155 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73) 2024-03-10T03:52:57.342 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:57 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:57.091+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.468226+0000 front 2024-03-10T03:52:02.468338+0000 (oldest deadline 2024-03-10T03:52:23.568069+0000) 2024-03-10T03:52:57.342 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:57 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:57.091+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:09.969210+0000 front 2024-03-10T03:52:09.969238+0000 (oldest deadline 2024-03-10T03:52:35.269187+0000) 2024-03-10T03:52:57.342 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:57 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:57.091+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:22.771171+0000 front 2024-03-10T03:52:22.771145+0000 (oldest deadline 2024-03-10T03:52:43.771599+0000) 2024-03-10T03:52:57.342 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:57 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6[40668]: 2024-03-10T03:52:57.091+0000 7fdbf9980700 -1 osd.6 73 heartbeat_check: no reply from 172.21.15.6:6830 osd.3 since back 2024-03-10T03:52:30.773431+0000 front 2024-03-10T03:52:30.773402+0000 (oldest deadline 2024-03-10T03:52:56.673313+0000) 2024-03-10T03:52:57.835 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:57 smithi071 podman[63291]: 2024-03-10 03:52:57.568189342 +0000 UTC m=+5.481907213 container died c25d08ba979da78ad6bdbc0c9f7e786f36b8adef5cc77b42bf4984351edbf5b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-10T03:52:58.091 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:57 smithi071 podman[63291]: 2024-03-10 03:52:57.991745936 +0000 UTC m=+5.905463810 container cleanup c25d08ba979da78ad6bdbc0c9f7e786f36b8adef5cc77b42bf4984351edbf5b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-10T03:52:58.091 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:57 smithi071 bash[63291]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6 2024-03-10T03:52:58.091 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:57 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:57.944+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:58.091 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:57 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:57.944+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:58.091 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:57 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:57.944+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:19.783838+0000 front 2024-03-10T03:52:19.783807+0000 (oldest deadline 2024-03-10T03:52:45.083769+0000) 2024-03-10T03:52:58.091 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:57 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:57.944+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6830 osd.3 since back 2024-03-10T03:52:30.384722+0000 front 2024-03-10T03:52:30.384735+0000 (oldest deadline 2024-03-10T03:52:52.084522+0000) 2024-03-10T03:52:58.091 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:57 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:57.944+0000 7faf0da5b700 -1 osd.7 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24424.0:155 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73) 2024-03-10T03:52:58.531 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:58 smithi071 podman[63415]: 2024-03-10 03:52:58.176218253 +0000 UTC m=+0.610894579 container remove c25d08ba979da78ad6bdbc0c9f7e786f36b8adef5cc77b42bf4984351edbf5b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-10T03:52:59.091 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:58.931+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:52:59.091 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:58.931+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:52:59.091 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:58.931+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:19.783838+0000 front 2024-03-10T03:52:19.783807+0000 (oldest deadline 2024-03-10T03:52:45.083769+0000) 2024-03-10T03:52:59.091 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:58.931+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6830 osd.3 since back 2024-03-10T03:52:30.384722+0000 front 2024-03-10T03:52:30.384735+0000 (oldest deadline 2024-03-10T03:52:52.084522+0000) 2024-03-10T03:52:59.091 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:58 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:58.931+0000 7faf0da5b700 -1 osd.7 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24424.0:155 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73) 2024-03-10T03:52:59.591 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:59 smithi071 podman[63542]: 2024-03-10 03:52:59.044296784 +0000 UTC m=+0.017783152 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada 2024-03-10T03:52:59.591 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:59 smithi071 podman[63542]: 2024-03-10 03:52:59.149006805 +0000 UTC m=+0.122493164 container create 210eb806e2c7332cd1e795682bd6c8d4ecefbff0cb964ad53bfe65fddb4d5511 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6-deactivate, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-10T03:52:59.591 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:59 smithi071 podman[63542]: 2024-03-10 03:52:59.326267779 +0000 UTC m=+0.299754150 container init 210eb806e2c7332cd1e795682bd6c8d4ecefbff0cb964ad53bfe65fddb4d5511 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222) 2024-03-10T03:52:59.591 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:59 smithi071 podman[63542]: 2024-03-10 03:52:59.329398992 +0000 UTC m=+0.302885352 container start 210eb806e2c7332cd1e795682bd6c8d4ecefbff0cb964ad53bfe65fddb4d5511 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-10T03:52:59.591 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:59 smithi071 podman[63542]: 2024-03-10 03:52:59.384472811 +0000 UTC m=+0.357959169 container attach 210eb806e2c7332cd1e795682bd6c8d4ecefbff0cb964ad53bfe65fddb4d5511 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-10T03:52:59.929 INFO:journalctl@ceph.osd.6.smithi071.stdout:Mar 10 03:52:59 smithi071 podman[63542]: 2024-03-10 03:52:59.631016559 +0000 UTC m=+0.604502920 container died 210eb806e2c7332cd1e795682bd6c8d4ecefbff0cb964ad53bfe65fddb4d5511 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.license=GPLv2) 2024-03-10T03:53:00.263 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:59.966+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:53:00.263 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:59.966+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:53:00.263 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:59.966+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:19.783838+0000 front 2024-03-10T03:52:19.783807+0000 (oldest deadline 2024-03-10T03:52:45.083769+0000) 2024-03-10T03:53:00.263 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:59.966+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6830 osd.3 since back 2024-03-10T03:52:30.384722+0000 front 2024-03-10T03:52:30.384735+0000 (oldest deadline 2024-03-10T03:52:52.084522+0000) 2024-03-10T03:53:00.263 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:52:59 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:52:59.966+0000 7faf0da5b700 -1 osd.7 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24424.0:155 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73) 2024-03-10T03:53:00.282 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.6.service' 2024-03-10T03:53:00.326 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T03:53:00.326 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-10T03:53:00.326 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-10T03:53:00.327 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.7 2024-03-10T03:53:00.590 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:00 smithi071 systemd[1]: Stopping Ceph osd.7 for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:53:00.930 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:00.615+0000 7faf11467700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T03:53:00.930 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:00.615+0000 7faf11467700 -1 osd.7 73 *** Got signal Terminated *** 2024-03-10T03:53:00.930 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:00.615+0000 7faf11467700 -1 osd.7 73 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-10T03:53:00.930 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:00.929+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:53:00.930 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:00.929+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:53:00.930 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:00.929+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:19.783838+0000 front 2024-03-10T03:52:19.783807+0000 (oldest deadline 2024-03-10T03:52:45.083769+0000) 2024-03-10T03:53:00.930 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:00.929+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6830 osd.3 since back 2024-03-10T03:52:30.384722+0000 front 2024-03-10T03:52:30.384735+0000 (oldest deadline 2024-03-10T03:52:52.084522+0000) 2024-03-10T03:53:00.931 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:00 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:00.929+0000 7faf0da5b700 -1 osd.7 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24424.0:155 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73) 2024-03-10T03:53:02.087 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:53:01 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[70283]: ts=2024-03-10T03:53:01.626Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi006.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.6:8443: connect: connection refused" 2024-03-10T03:53:02.087 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:53:01 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[70283]: ts=2024-03-10T03:53:01.627Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi071.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.71:8443: connect: connection refused" 2024-03-10T03:53:02.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:01 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:01.907+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:53:02.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:01 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:01.907+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:53:02.342 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:01 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:01.907+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:19.783838+0000 front 2024-03-10T03:52:19.783807+0000 (oldest deadline 2024-03-10T03:52:45.083769+0000) 2024-03-10T03:53:02.342 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:01 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:01.907+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6830 osd.3 since back 2024-03-10T03:52:30.384722+0000 front 2024-03-10T03:52:30.384735+0000 (oldest deadline 2024-03-10T03:52:52.084522+0000) 2024-03-10T03:53:02.342 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:01 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:01.907+0000 7faf0da5b700 -1 osd.7 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24424.0:155 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73) 2024-03-10T03:53:03.342 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:02 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:02.919+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:53:03.342 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:02 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:02.919+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:53:03.342 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:02 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:02.919+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:19.783838+0000 front 2024-03-10T03:52:19.783807+0000 (oldest deadline 2024-03-10T03:52:45.083769+0000) 2024-03-10T03:53:03.342 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:02 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:02.919+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6830 osd.3 since back 2024-03-10T03:52:30.384722+0000 front 2024-03-10T03:52:30.384735+0000 (oldest deadline 2024-03-10T03:52:52.084522+0000) 2024-03-10T03:53:03.343 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:02 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:02.919+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-03-10T03:52:38.485717+0000 front 2024-03-10T03:52:38.485746+0000 (oldest deadline 2024-03-10T03:53:02.585826+0000) 2024-03-10T03:53:03.343 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:02 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:02.919+0000 7faf0da5b700 -1 osd.7 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24424.0:155 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73) 2024-03-10T03:53:04.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:03 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:03.918+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:53:04.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:03 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:03.918+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:53:04.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:03 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:03.918+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:19.783838+0000 front 2024-03-10T03:52:19.783807+0000 (oldest deadline 2024-03-10T03:52:45.083769+0000) 2024-03-10T03:53:04.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:03 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:03.918+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6830 osd.3 since back 2024-03-10T03:52:30.384722+0000 front 2024-03-10T03:52:30.384735+0000 (oldest deadline 2024-03-10T03:52:52.084522+0000) 2024-03-10T03:53:04.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:03 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:03.918+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-03-10T03:52:38.485717+0000 front 2024-03-10T03:52:38.485746+0000 (oldest deadline 2024-03-10T03:53:02.585826+0000) 2024-03-10T03:53:04.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:03 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:03.918+0000 7faf0da5b700 -1 osd.7 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24424.0:155 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73) 2024-03-10T03:53:05.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:04 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:04.920+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6806 osd.0 since back 2024-03-10T03:52:02.782318+0000 front 2024-03-10T03:52:02.782295+0000 (oldest deadline 2024-03-10T03:52:26.882143+0000) 2024-03-10T03:53:05.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:04 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:04.920+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6814 osd.1 since back 2024-03-10T03:52:06.882755+0000 front 2024-03-10T03:52:06.882708+0000 (oldest deadline 2024-03-10T03:52:32.182574+0000) 2024-03-10T03:53:05.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:04 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:04.920+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6822 osd.2 since back 2024-03-10T03:52:19.783838+0000 front 2024-03-10T03:52:19.783807+0000 (oldest deadline 2024-03-10T03:52:45.083769+0000) 2024-03-10T03:53:05.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:04 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:04.920+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.6:6830 osd.3 since back 2024-03-10T03:52:30.384722+0000 front 2024-03-10T03:52:30.384735+0000 (oldest deadline 2024-03-10T03:52:52.084522+0000) 2024-03-10T03:53:05.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:04 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:04.920+0000 7faf0da5b700 -1 osd.7 73 heartbeat_check: no reply from 172.21.15.71:6804 osd.4 since back 2024-03-10T03:52:38.485717+0000 front 2024-03-10T03:52:38.485746+0000 (oldest deadline 2024-03-10T03:53:02.585826+0000) 2024-03-10T03:53:05.341 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:04 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7[44244]: 2024-03-10T03:53:04.920+0000 7faf0da5b700 -1 osd.7 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24424.0:155 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e73) 2024-03-10T03:53:06.340 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:06 smithi071 podman[63648]: 2024-03-10 03:53:06.114202293 +0000 UTC m=+5.610708856 container died 0541b9131538080580abc7a1a896420e4014a0a0812d0065c15306336f1c2660 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-10T03:53:06.666 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:06 smithi071 podman[63648]: 2024-03-10 03:53:06.498592799 +0000 UTC m=+5.995099418 container cleanup 0541b9131538080580abc7a1a896420e4014a0a0812d0065c15306336f1c2660 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) 2024-03-10T03:53:06.666 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:06 smithi071 bash[63648]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7 2024-03-10T03:53:07.090 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:06 smithi071 podman[63703]: 2024-03-10 03:53:06.666189089 +0000 UTC m=+0.553407456 container remove 0541b9131538080580abc7a1a896420e4014a0a0812d0065c15306336f1c2660 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18) 2024-03-10T03:53:07.420 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:07 smithi071 podman[63902]: 2024-03-10 03:53:07.019464996 +0000 UTC m=+0.020005041 image pull 01a4f6a6bb3e92ab9ab39a2f835684ea9b10e6ec9fb81ac08ebea4165583a538 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada 2024-03-10T03:53:07.420 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:07 smithi071 podman[63902]: 2024-03-10 03:53:07.126419947 +0000 UTC m=+0.126959973 container create dbe5c0f9e60f54c653d063840ac7dc3c192a2f40c112c0399744478350c2c81f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-10T03:53:07.420 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:07 smithi071 podman[63902]: 2024-03-10 03:53:07.335806737 +0000 UTC m=+0.336346768 container init dbe5c0f9e60f54c653d063840ac7dc3c192a2f40c112c0399744478350c2c81f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_CLEAN=True, org.label-schema.build-date=20240222) 2024-03-10T03:53:07.420 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:07 smithi071 podman[63902]: 2024-03-10 03:53:07.338857993 +0000 UTC m=+0.339398022 container start dbe5c0f9e60f54c653d063840ac7dc3c192a2f40c112c0399744478350c2c81f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7-deactivate, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-10T03:53:07.420 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:07 smithi071 podman[63902]: 2024-03-10 03:53:07.419260071 +0000 UTC m=+0.419800104 container attach dbe5c0f9e60f54c653d063840ac7dc3c192a2f40c112c0399744478350c2c81f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.build-date=20240222, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-10T03:53:07.841 INFO:journalctl@ceph.osd.7.smithi071.stdout:Mar 10 03:53:07 smithi071 podman[63902]: 2024-03-10 03:53:07.64599053 +0000 UTC m=+0.646530568 container died dbe5c0f9e60f54c653d063840ac7dc3c192a2f40c112c0399744478350c2c81f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-osd-7-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2) 2024-03-10T03:53:08.286 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@osd.7.service' 2024-03-10T03:53:08.332 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T03:53:08.333 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-10T03:53:08.333 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-10T03:53:08.333 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-414ba1e0-de90-11ee-95c7-87774f69a715@rgw.foo.a 2024-03-10T03:53:08.662 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 10 03:53:08 smithi006 systemd[1]: Stopping Ceph rgw.foo.a for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:53:09.086 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 10 03:53:08 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-rgw-foo-a[50825]: 2024-03-10T03:53:08.657+0000 7fe137962700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-10T03:53:09.086 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 10 03:53:08 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-rgw-foo-a[50825]: 2024-03-10T03:53:08.657+0000 7fe1461daac0 -1 shutting down 2024-03-10T03:53:11.716 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:53:11 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[70283]: ts=2024-03-10T03:53:11.626Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi071.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.71:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi006.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.6:8443: connect: connection refused" 2024-03-10T03:53:11.716 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:53:11 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[70283]: ts=2024-03-10T03:53:11.626Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi006.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.6:8443: connect: connection refused" 2024-03-10T03:53:11.716 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:53:11 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[70283]: ts=2024-03-10T03:53:11.627Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi071.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.71:8443: connect: connection refused" 2024-03-10T03:53:18.940 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 10 03:53:18 smithi006 bash[74526]: time="2024-03-10T03:53:18Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-414ba1e0-de90-11ee-95c7-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-10T03:53:18.940 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 10 03:53:18 smithi006 podman[74526]: 2024-03-10 03:53:18.678535551 +0000 UTC m=+10.140904905 container died 379f2bdd26efd6ca75e9b305be6b2a0ea2d64e7684c61d6f92cbca5ea2845ad6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-rgw-foo-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222) 2024-03-10T03:53:19.197 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 10 03:53:19 smithi006 podman[74526]: 2024-03-10 03:53:19.013365193 +0000 UTC m=+10.475734544 container cleanup 379f2bdd26efd6ca75e9b305be6b2a0ea2d64e7684c61d6f92cbca5ea2845ad6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-rgw-foo-a, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-10T03:53:19.198 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 10 03:53:19 smithi006 bash[74526]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715-rgw-foo-a 2024-03-10T03:53:19.449 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@rgw.foo.a.service' 2024-03-10T03:53:19.474 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 10 03:53:19 smithi006 podman[75280]: 2024-03-10 03:53:19.197779971 +0000 UTC m=+0.513597717 container remove 379f2bdd26efd6ca75e9b305be6b2a0ea2d64e7684c61d6f92cbca5ea2845ad6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-rgw-foo-a, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-10T03:53:19.474 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 10 03:53:19 smithi006 systemd[1]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-10T03:53:19.474 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 10 03:53:19 smithi006 systemd[1]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715@rgw.foo.a.service: Failed with result 'exit-code'. 2024-03-10T03:53:19.474 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 10 03:53:19 smithi006 systemd[1]: Stopped Ceph rgw.foo.a for 414ba1e0-de90-11ee-95c7-87774f69a715. 2024-03-10T03:53:19.474 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 10 03:53:19 smithi006 systemd[1]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715@rgw.foo.a.service: Consumed 1.713s CPU time. 2024-03-10T03:53:19.488 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T03:53:19.489 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-10T03:53:19.489 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-10T03:53:19.489 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-414ba1e0-de90-11ee-95c7-87774f69a715@prometheus.a 2024-03-10T03:53:19.796 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:53:19 smithi071 systemd[1]: Stopping Ceph prometheus.a for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:53:20.072 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:53:19 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:53:19.796Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-10T03:53:20.072 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:53:19 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:53:19.796Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-10T03:53:20.072 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:53:19 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:53:19.796Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-10T03:53:20.072 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:53:19 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:53:19.796Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-10T03:53:20.072 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:53:19 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:53:19.796Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-10T03:53:20.072 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:53:19 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:53:19.796Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-10T03:53:20.072 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:53:19 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:53:19.796Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-10T03:53:20.072 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:53:19 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:53:19.796Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-10T03:53:20.073 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:53:19 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:53:19.796Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-10T03:53:20.073 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:53:19 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:53:19.797Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-10T03:53:20.073 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:53:19 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:53:19.797Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-10T03:53:20.073 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:53:19 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a[60071]: ts=2024-03-10T03:53:19.797Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-10T03:53:20.073 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:53:19 smithi071 podman[64675]: 2024-03-10 03:53:19.806850808 +0000 UTC m=+0.124141160 container died 1b2d5f2bc0817a3227b106a1cff99e392a034a9d497b555145d4d9d8184e0421 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-10T03:53:20.340 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:53:20 smithi071 podman[64675]: 2024-03-10 03:53:20.162768641 +0000 UTC m=+0.480058984 container cleanup 1b2d5f2bc0817a3227b106a1cff99e392a034a9d497b555145d4d9d8184e0421 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-10T03:53:20.341 INFO:journalctl@ceph.prometheus.a.smithi071.stdout:Mar 10 03:53:20 smithi071 bash[64675]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715-prometheus-a 2024-03-10T03:53:20.512 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-414ba1e0-de90-11ee-95c7-87774f69a715@prometheus.a.service' 2024-03-10T03:53:20.551 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-10T03:53:20.551 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-10T03:53:20.552 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 --force --keep-logs 2024-03-10T03:53:20.747 INFO:teuthology.orchestra.run.smithi006.stdout:Deleting cluster with fsid: 414ba1e0-de90-11ee-95c7-87774f69a715 2024-03-10T03:53:22.035 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:53:21 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[70283]: ts=2024-03-10T03:53:21.627Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi006.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.6:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi071.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.71:8443: connect: connection refused" 2024-03-10T03:53:22.036 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:53:21 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[70283]: ts=2024-03-10T03:53:21.628Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi006.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.6:8443: connect: connection refused" 2024-03-10T03:53:22.036 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:53:21 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[70283]: ts=2024-03-10T03:53:21.628Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi071.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.71:8443: connect: connection refused" 2024-03-10T03:53:22.836 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:53:22 smithi006 systemd[1]: Stopping Ceph alertmanager.a for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:53:23.120 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:53:22 smithi006 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a[70283]: ts=2024-03-10T03:53:22.851Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-10T03:53:23.120 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:53:22 smithi006 podman[75798]: 2024-03-10 03:53:22.872245098 +0000 UTC m=+0.116274893 container died 1b85b458da5b874b97d990604127deb2444303c1e0471ba5b6e8e1b9942bae07 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-10T03:53:23.411 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:53:23 smithi006 podman[75798]: 2024-03-10 03:53:23.218460848 +0000 UTC m=+0.462490651 container cleanup 1b85b458da5b874b97d990604127deb2444303c1e0471ba5b6e8e1b9942bae07 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-10T03:53:23.411 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:53:23 smithi006 bash[75798]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a 2024-03-10T03:53:23.785 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:53:23 smithi006 podman[75812]: 2024-03-10 03:53:23.411278019 +0000 UTC m=+0.534354203 container remove 1b85b458da5b874b97d990604127deb2444303c1e0471ba5b6e8e1b9942bae07 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-10T03:53:23.785 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:53:23 smithi006 podman[75812]: 2024-03-10 03:53:23.495314031 +0000 UTC m=+0.618390226 volume remove 65de3704eda0f3f9daf2f66a88648764865711e7f92a1b132026912593369491 2024-03-10T03:53:24.086 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:53:24 smithi006 systemd[1]: Stopping Ceph node-exporter.a for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:53:24.086 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:53:23 smithi006 systemd[1]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-10T03:53:24.086 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 10 03:53:23 smithi006 systemd[1]: Stopped Ceph alertmanager.a for 414ba1e0-de90-11ee-95c7-87774f69a715. 2024-03-10T03:53:24.586 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:53:24 smithi006 podman[75923]: 2024-03-10 03:53:24.291940984 +0000 UTC m=+0.134210385 container died 35152e971ef48bfd9af588dca34d6e25af3f38185241106ca353b6536b1a6f56 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-10T03:53:25.020 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:53:24 smithi006 podman[75923]: 2024-03-10 03:53:24.622781747 +0000 UTC m=+0.465051145 container cleanup 35152e971ef48bfd9af588dca34d6e25af3f38185241106ca353b6536b1a6f56 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-10T03:53:25.020 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:53:24 smithi006 bash[75923]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a 2024-03-10T03:53:25.020 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:53:24 smithi006 podman[75937]: 2024-03-10 03:53:24.798898097 +0000 UTC m=+0.508031375 container remove 35152e971ef48bfd9af588dca34d6e25af3f38185241106ca353b6536b1a6f56 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-10T03:53:25.020 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:53:24 smithi006 systemd[1]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-10T03:53:25.336 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:53:25 smithi006 systemd[1]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-10T03:53:25.336 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:53:25 smithi006 systemd[1]: Stopped Ceph node-exporter.a for 414ba1e0-de90-11ee-95c7-87774f69a715. 2024-03-10T03:53:25.336 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 10 03:53:25 smithi006 systemd[1]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715@node-exporter.a.service: Consumed 1.877s CPU time. 2024-03-10T03:53:25.691 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 --force --keep-logs 2024-03-10T03:53:25.880 INFO:teuthology.orchestra.run.smithi071.stdout:Deleting cluster with fsid: 414ba1e0-de90-11ee-95c7-87774f69a715 2024-03-10T03:53:27.591 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:53:27 smithi071 systemd[1]: Stopping Ceph iscsi.iscsi.a for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:53:28.090 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:53:27 smithi071 ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a[47087]: debug Shutdown received 2024-03-10T03:53:37.990 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:53:37 smithi071 bash[65129]: time="2024-03-10T03:53:37Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-10T03:53:37.991 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:53:37 smithi071 podman[65129]: 2024-03-10 03:53:37.731065124 +0000 UTC m=+10.137004677 container died ce51f431d58c2c08ee8cea646575007a5201092d9c58f1071a9b2f51ac37cdd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, GIT_BRANCH=HEAD) 2024-03-10T03:53:38.340 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:53:38 smithi071 podman[65129]: 2024-03-10 03:53:38.139515903 +0000 UTC m=+10.545455456 container cleanup ce51f431d58c2c08ee8cea646575007a5201092d9c58f1071a9b2f51ac37cdd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux ) 2024-03-10T03:53:38.341 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:53:38 smithi071 bash[65129]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a 2024-03-10T03:53:38.777 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:53:38 smithi071 podman[65144]: 2024-03-10 03:53:38.348749799 +0000 UTC m=+0.616055517 container remove ce51f431d58c2c08ee8cea646575007a5201092d9c58f1071a9b2f51ac37cdd8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bc4a24690ce5d7255d490e21058d8c114b55e1b93a3d8345f90b6700ad29dada, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-iscsi-iscsi-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-yuri4-testing-2024-03-05-0854-831bc18, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-10T03:53:38.777 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:53:38 smithi071 systemd[1]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-10T03:53:38.777 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:53:38 smithi071 systemd[1]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-10T03:53:38.777 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:53:38 smithi071 systemd[1]: Stopped Ceph iscsi.iscsi.a for 414ba1e0-de90-11ee-95c7-87774f69a715. 2024-03-10T03:53:38.777 INFO:journalctl@ceph.iscsi.iscsi.a.smithi071.stdout:Mar 10 03:53:38 smithi071 systemd[1]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715@iscsi.iscsi.a.service: Consumed 2.938s CPU time. 2024-03-10T03:53:40.090 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:53:39 smithi071 systemd[1]: Stopping Ceph grafana.a for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:53:40.590 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:53:40 smithi071 podman[65402]: 2024-03-10 03:53:40.198927798 +0000 UTC m=+0.104646812 container died 1c68ee40279e3da4885ebd6774b38dea2803009cf2f157afd3e659d4e9ab3882 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, distribution-scope=public, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, name=ubi8, vcs-type=git, version=8.5, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, release=239.1651231664) 2024-03-10T03:53:40.926 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:53:40 smithi071 podman[65402]: 2024-03-10 03:53:40.604601428 +0000 UTC m=+0.510320432 container cleanup 1c68ee40279e3da4885ebd6774b38dea2803009cf2f157afd3e659d4e9ab3882 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, architecture=x86_64, io.buildah.version=1.32.0, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, build-date=2022-04-29T11:48:11.055897, com.redhat.component=ubi8-container) 2024-03-10T03:53:40.926 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:53:40 smithi071 bash[65402]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a 2024-03-10T03:53:40.926 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:53:40 smithi071 podman[65416]: 2024-03-10 03:53:40.814403648 +0000 UTC m=+0.618318207 container remove 1c68ee40279e3da4885ebd6774b38dea2803009cf2f157afd3e659d4e9ab3882 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, description=Ceph Grafana Container, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, vcs-type=git, name=ubi8, release=239.1651231664, maintainer=Paul Cuzner , version=8.5, build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., io.buildah.version=1.32.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-10T03:53:40.926 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:53:40 smithi071 systemd[1]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-10T03:53:41.340 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:53:41 smithi071 systemd[1]: Stopping Ceph node-exporter.b for 414ba1e0-de90-11ee-95c7-87774f69a715... 2024-03-10T03:53:41.341 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:53:40 smithi071 systemd[1]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-10T03:53:41.341 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:53:40 smithi071 systemd[1]: Stopped Ceph grafana.a for 414ba1e0-de90-11ee-95c7-87774f69a715. 2024-03-10T03:53:41.341 INFO:journalctl@ceph.grafana.a.smithi071.stdout:Mar 10 03:53:40 smithi071 systemd[1]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715@grafana.a.service: Consumed 4.733s CPU time. 2024-03-10T03:53:41.650 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:53:41 smithi071 podman[65527]: 2024-03-10 03:53:41.39454584 +0000 UTC m=+0.117534544 container died b8040739857f947aa2ca5a899c8722897eb3053274d92a18a9dcff5f68090cbe (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-10T03:53:41.909 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:53:41 smithi071 podman[65527]: 2024-03-10 03:53:41.725106899 +0000 UTC m=+0.448095601 container cleanup b8040739857f947aa2ca5a899c8722897eb3053274d92a18a9dcff5f68090cbe (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-10T03:53:41.909 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:53:41 smithi071 bash[65527]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b 2024-03-10T03:53:42.290 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:53:41 smithi071 podman[65541]: 2024-03-10 03:53:41.909316596 +0000 UTC m=+0.519325424 container remove b8040739857f947aa2ca5a899c8722897eb3053274d92a18a9dcff5f68090cbe (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-414ba1e0-de90-11ee-95c7-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-10T03:53:42.291 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:53:41 smithi071 systemd[1]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-10T03:53:42.291 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:53:42 smithi071 systemd[1]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-10T03:53:42.291 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:53:42 smithi071 systemd[1]: Stopped Ceph node-exporter.b for 414ba1e0-de90-11ee-95c7-87774f69a715. 2024-03-10T03:53:42.291 INFO:journalctl@ceph.node-exporter.b.smithi071.stdout:Mar 10 03:53:42 smithi071 systemd[1]: ceph-414ba1e0-de90-11ee-95c7-87774f69a715@node-exporter.b.service: Consumed 1.818s CPU time. 2024-03-10T03:53:42.842 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-10T03:53:42.877 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-10T03:53:42.907 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-10T03:53:42.908 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587949/remote/smithi006/crash 2024-03-10T03:53:42.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/crash -- . 2024-03-10T03:53:42.962 INFO:teuthology.orchestra.run.smithi006.stderr:tar: /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/crash: Cannot open: No such file or directory 2024-03-10T03:53:42.963 INFO:teuthology.orchestra.run.smithi006.stderr:tar: Error is not recoverable: exiting now 2024-03-10T03:53:42.964 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587949/remote/smithi071/crash 2024-03-10T03:53:42.965 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/crash -- . 2024-03-10T03:53:43.005 INFO:teuthology.orchestra.run.smithi071.stderr:tar: /var/lib/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/crash: Cannot open: No such file or directory 2024-03-10T03:53:43.006 INFO:teuthology.orchestra.run.smithi071.stderr:tar: Error is not recoverable: exiting now 2024-03-10T03:53:43.007 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-10T03:53:43.007 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | head -n 1 2024-03-10T03:53:43.038 INFO:teuthology.orchestra.run.smithi006.stdout:2024-03-10T03:45:23.518476+0000 mon.a (mon.0) 212 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-03-10T03:53:43.038 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-10T03:53:43.038 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[SEC\]' /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | head -n 1 2024-03-10T03:53:43.108 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[ERR\]' /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | head -n 1 2024-03-10T03:53:43.178 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[WRN\]' /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | head -n 1 2024-03-10T03:53:43.248 INFO:teuthology.orchestra.run.smithi006.stdout:2024-03-10T03:45:23.518476+0000 mon.a (mon.0) 212 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-03-10T03:53:43.248 INFO:tasks.cephadm:Compressing logs... 2024-03-10T03:53:43.248 DEBUG:teuthology.orchestra.run.smithi006:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-10T03:53:43.290 DEBUG:teuthology.orchestra.run.smithi071:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-10T03:53:43.314 INFO:teuthology.orchestra.run.smithi006.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-10T03:53:43.314 INFO:teuthology.orchestra.run.smithi006.stderr:: No such file or directory 2024-03-10T03:53:43.314 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-mon.a.log 2024-03-10T03:53:43.315 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph.audit.log 2024-03-10T03:53:43.316 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-10T03:53:43.316 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-mon.a.log: /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-osd.1.log 2024-03-10T03:53:43.316 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-osd.3.log 2024-03-10T03:53:43.316 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-osd.1.log: 57.6% -- replaced with /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-10T03:53:43.329 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-mgr.y.log 2024-03-10T03:53:43.329 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-osd.0.log 2024-03-10T03:53:43.329 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph.cephadm.log 2024-03-10T03:53:43.329 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-osd.0.log: 93.9% 86.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-10T03:53:43.329 INFO:teuthology.orchestra.run.smithi006.stderr: -- replaced with /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph.audit.log.gz 2024-03-10T03:53:43.329 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph.log 2024-03-10T03:53:43.330 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-osd.2.log 2024-03-10T03:53:43.330 INFO:teuthology.orchestra.run.smithi006.stderr: 88.8% -- replaced with /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph.cephadm.log.gz 2024-03-10T03:53:43.330 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph.log: 91.8% -- replaced with /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph.log.gz 2024-03-10T03:53:43.330 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-volume.log 2024-03-10T03:53:43.330 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-mon.c.log 2024-03-10T03:53:43.334 INFO:teuthology.orchestra.run.smithi071.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-10T03:53:43.334 INFO:teuthology.orchestra.run.smithi071.stderr:: No such file or directory 2024-03-10T03:53:43.335 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-osd.6.log 2024-03-10T03:53:43.335 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph.audit.log 2024-03-10T03:53:43.335 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-osd.4.log 2024-03-10T03:53:43.335 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-osd.7.log 2024-03-10T03:53:43.336 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-osd.5.log 2024-03-10T03:53:43.336 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-mgr.x.log 2024-03-10T03:53:43.336 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/tcmu-runner.log 2024-03-10T03:53:43.336 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph.cephadm.log 2024-03-10T03:53:43.336 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/tcmu-runner.log: 90.3% -- replaced with /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph.audit.log.gz 2024-03-10T03:53:43.336 INFO:teuthology.orchestra.run.smithi071.stderr: 67.6% -- replaced with /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/tcmu-runner.log.gz 2024-03-10T03:53:43.337 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph.log 2024-03-10T03:53:43.337 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-volume.log 2024-03-10T03:53:43.337 INFO:teuthology.orchestra.run.smithi071.stderr: 81.1% -- replaced with /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph.cephadm.log.gz 2024-03-10T03:53:43.337 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph.log: 86.0% -- replaced with /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph.log.gz 2024-03-10T03:53:43.337 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-mon.b.log 2024-03-10T03:53:43.337 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-volume.log: /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-mon.b.log: 82.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-10T03:53:43.338 INFO:teuthology.orchestra.run.smithi071.stderr: 90.8% -- replaced with /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-mgr.x.log.gz 2024-03-10T03:53:43.361 INFO:teuthology.orchestra.run.smithi071.stderr: 90.2% -- replaced with /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-volume.log.gz 2024-03-10T03:53:43.384 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-volume.log: /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-mon.c.log: 90.1% -- replaced with /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-volume.log.gz 2024-03-10T03:53:43.488 INFO:teuthology.orchestra.run.smithi006.stderr: 88.1% -- replaced with /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-mgr.y.log.gz 2024-03-10T03:53:43.497 INFO:teuthology.orchestra.run.smithi071.stderr: 92.1% -- replaced with /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-mon.b.log.gz 2024-03-10T03:53:43.531 INFO:teuthology.orchestra.run.smithi006.stderr: 92.3% -- replaced with /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-mon.c.log.gz 2024-03-10T03:53:43.532 INFO:teuthology.orchestra.run.smithi006.stderr: 92.9% -- replaced with /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-osd.2.log.gz 2024-03-10T03:53:43.542 INFO:teuthology.orchestra.run.smithi071.stderr: 93.1% -- replaced with /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-osd.5.log.gz 2024-03-10T03:53:43.544 INFO:teuthology.orchestra.run.smithi071.stderr: 93.0% -- replaced with /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-osd.6.log.gz 2024-03-10T03:53:43.554 INFO:teuthology.orchestra.run.smithi071.stderr: 93.4% -- replaced with /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-osd.4.log.gz 2024-03-10T03:53:43.562 INFO:teuthology.orchestra.run.smithi071.stderr: 93.1% -- replaced with /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-osd.7.log.gz 2024-03-10T03:53:43.564 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-03-10T03:53:43.565 INFO:teuthology.orchestra.run.smithi071.stderr:real 0m0.260s 2024-03-10T03:53:43.565 INFO:teuthology.orchestra.run.smithi071.stderr:user 0m1.132s 2024-03-10T03:53:43.565 INFO:teuthology.orchestra.run.smithi071.stderr:sys 0m0.062s 2024-03-10T03:53:43.641 INFO:teuthology.orchestra.run.smithi006.stderr: 93.5% -- replaced with /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-osd.3.log.gz 2024-03-10T03:53:43.649 INFO:teuthology.orchestra.run.smithi006.stderr: 93.4% -- replaced with /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-osd.0.log.gz 2024-03-10T03:53:43.694 INFO:teuthology.orchestra.run.smithi006.stderr: 93.5% -- replaced with /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-osd.1.log.gz 2024-03-10T03:53:43.771 INFO:teuthology.orchestra.run.smithi006.stderr: 91.2% -- replaced with /var/log/ceph/414ba1e0-de90-11ee-95c7-87774f69a715/ceph-mon.a.log.gz 2024-03-10T03:53:43.772 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-10T03:53:43.773 INFO:teuthology.orchestra.run.smithi006.stderr:real 0m0.472s 2024-03-10T03:53:43.773 INFO:teuthology.orchestra.run.smithi006.stderr:user 0m2.024s 2024-03-10T03:53:43.773 INFO:teuthology.orchestra.run.smithi006.stderr:sys 0m0.092s 2024-03-10T03:53:43.773 INFO:tasks.cephadm:Archiving logs... 2024-03-10T03:53:43.773 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587949/remote/smithi006/log 2024-03-10T03:53:43.774 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-10T03:53:44.111 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587949/remote/smithi071/log 2024-03-10T03:53:44.112 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-10T03:53:44.297 INFO:tasks.cephadm:Removing cluster... 2024-03-10T03:53:44.297 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 --force 2024-03-10T03:53:44.481 INFO:teuthology.orchestra.run.smithi006.stdout:Deleting cluster with fsid: 414ba1e0-de90-11ee-95c7-87774f69a715 2024-03-10T03:53:44.604 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 414ba1e0-de90-11ee-95c7-87774f69a715 --force 2024-03-10T03:53:44.782 INFO:teuthology.orchestra.run.smithi071.stdout:Deleting cluster with fsid: 414ba1e0-de90-11ee-95c7-87774f69a715 2024-03-10T03:53:44.900 INFO:tasks.cephadm:Removing cephadm ... 2024-03-10T03:53:44.900 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-10T03:53:44.914 DEBUG:teuthology.orchestra.run.smithi071:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-10T03:53:44.930 INFO:tasks.cephadm:Teardown complete 2024-03-10T03:53:44.930 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-03-10T03:53:44.947 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi006:/dev/vg_nvme/lv_1... 2024-03-10T03:53:44.948 DEBUG:teuthology.orchestra.run.smithi006:> sudo nvme disconnect -n lv_1 2024-03-10T03:53:45.110 INFO:teuthology.orchestra.run.smithi006.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-10T03:53:45.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T03:53:45.112 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi006:/dev/vg_nvme/lv_2... 2024-03-10T03:53:45.112 DEBUG:teuthology.orchestra.run.smithi006:> sudo nvme disconnect -n lv_2 2024-03-10T03:53:45.254 INFO:teuthology.orchestra.run.smithi006.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-10T03:53:45.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T03:53:45.256 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi006:/dev/vg_nvme/lv_3... 2024-03-10T03:53:45.256 DEBUG:teuthology.orchestra.run.smithi006:> sudo nvme disconnect -n lv_3 2024-03-10T03:53:45.396 INFO:teuthology.orchestra.run.smithi006.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-10T03:53:45.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T03:53:45.398 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi006:/dev/vg_nvme/lv_4... 2024-03-10T03:53:45.398 DEBUG:teuthology.orchestra.run.smithi006:> sudo nvme disconnect -n lv_4 2024-03-10T03:53:45.539 INFO:teuthology.orchestra.run.smithi006.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-10T03:53:45.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T03:53:45.541 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-10T03:53:45.541 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/scratch_devs 2024-03-10T03:53:45.567 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi071:/dev/vg_nvme/lv_1... 2024-03-10T03:53:45.568 DEBUG:teuthology.orchestra.run.smithi071:> sudo nvme disconnect -n lv_1 2024-03-10T03:53:45.710 INFO:teuthology.orchestra.run.smithi071.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-10T03:53:45.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T03:53:45.711 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi071:/dev/vg_nvme/lv_2... 2024-03-10T03:53:45.712 DEBUG:teuthology.orchestra.run.smithi071:> sudo nvme disconnect -n lv_2 2024-03-10T03:53:45.878 INFO:teuthology.orchestra.run.smithi071.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-10T03:53:45.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T03:53:45.879 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi071:/dev/vg_nvme/lv_3... 2024-03-10T03:53:45.879 DEBUG:teuthology.orchestra.run.smithi071:> sudo nvme disconnect -n lv_3 2024-03-10T03:53:46.055 INFO:teuthology.orchestra.run.smithi071.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-10T03:53:46.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T03:53:46.056 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi071:/dev/vg_nvme/lv_4... 2024-03-10T03:53:46.056 DEBUG:teuthology.orchestra.run.smithi071:> sudo nvme disconnect -n lv_4 2024-03-10T03:53:46.228 INFO:teuthology.orchestra.run.smithi071.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-10T03:53:46.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T03:53:46.230 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-03-10T03:53:46.230 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/scratch_devs 2024-03-10T03:53:46.257 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-10T03:53:46.266 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-10T03:53:46.266 DEBUG:teuthology.orchestra.run.smithi006:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-10T03:53:46.269 DEBUG:teuthology.orchestra.run.smithi071:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-10T03:53:46.282 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2024-03-10T03:53:46.288 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-10T03:53:46.288 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2024-03-10T03:53:46.288 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 12 +855us[ +863us] +/- 118ms 2024-03-10T03:53:46.288 INFO:teuthology.orchestra.run.smithi006.stdout:^* hv02.front.sepia.ceph.com 3 6 377 11 -520us[ -513us] +/- 56ms 2024-03-10T03:53:46.288 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 11 +172us[ +172us] +/- 74ms 2024-03-10T03:53:46.288 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-10T03:53:46.311 INFO:teuthology.orchestra.run.smithi071.stderr:bash: line 1: ntpq: command not found 2024-03-10T03:53:46.315 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-10T03:53:46.315 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2024-03-10T03:53:46.315 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 13 +860us[ +852us] +/- 118ms 2024-03-10T03:53:46.315 INFO:teuthology.orchestra.run.smithi071.stdout:^* hv02.front.sepia.ceph.com 3 6 377 12 -522us[ -531us] +/- 56ms 2024-03-10T03:53:46.315 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 13 +172us[ +163us] +/- 74ms 2024-03-10T03:53:46.315 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-10T03:53:46.316 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-10T03:53:46.328 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-10T03:53:46.329 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-10T03:53:46.356 DEBUG:teuthology.orchestra.run.smithi006:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-10T03:53:46.435 DEBUG:teuthology.orchestra.run.smithi071:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-10T03:53:46.506 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-10T03:53:46.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T03:53:46.537 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-10T03:53:46.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T03:53:46.567 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-10T03:53:46.577 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-10T03:53:46.597 INFO:teuthology.task.internal:Duration was 919.465131 seconds 2024-03-10T03:53:46.597 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-10T03:53:46.630 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-10T03:53:46.630 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-10T03:53:46.633 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-10T03:53:46.675 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-10T03:53:46.676 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-10T03:53:47.132 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-10T03:53:47.132 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi006.front.sepia.ceph.com 2024-03-10T03:53:47.132 DEBUG:teuthology.orchestra.run.smithi006:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-10T03:53:47.157 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi071.front.sepia.ceph.com 2024-03-10T03:53:47.158 DEBUG:teuthology.orchestra.run.smithi071:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-10T03:53:47.189 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-10T03:53:47.189 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-10T03:53:47.198 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-10T03:53:47.257 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-10T03:53:47.258 DEBUG:teuthology.orchestra.run.smithi006:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-10T03:53:47.268 DEBUG:teuthology.orchestra.run.smithi071:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-10T03:53:47.537 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-10T03:53:47.547 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-10T03:53:47.547 DEBUG:teuthology.orchestra.run.smithi006:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-10T03:53:47.574 DEBUG:teuthology.orchestra.run.smithi071:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-10T03:53:47.601 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-10T03:53:47.613 DEBUG:teuthology.orchestra.run.smithi006:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-10T03:53:47.617 DEBUG:teuthology.orchestra.run.smithi071:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-10T03:53:47.639 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = core 2024-03-10T03:53:47.666 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = core 2024-03-10T03:53:47.686 DEBUG:teuthology.orchestra.run.smithi006:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-10T03:53:47.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T03:53:47.738 DEBUG:teuthology.orchestra.run.smithi071:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-10T03:53:47.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-10T03:53:47.753 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-10T03:53:47.765 INFO:teuthology.task.internal:Transferring archived files... 2024-03-10T03:53:47.766 DEBUG:teuthology.misc:Transferring archived files from smithi006:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587949/remote/smithi006 2024-03-10T03:53:47.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-10T03:53:47.821 DEBUG:teuthology.misc:Transferring archived files from smithi071:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587949/remote/smithi071 2024-03-10T03:53:47.821 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-10T03:53:47.865 INFO:teuthology.task.internal:Removing archive directory... 2024-03-10T03:53:47.865 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-10T03:53:47.868 DEBUG:teuthology.orchestra.run.smithi071:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-10T03:53:47.912 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-10T03:53:47.924 INFO:teuthology.task.internal:Not uploading archives. 2024-03-10T03:53:47.924 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-10T03:53:47.956 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-10T03:53:47.956 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-10T03:53:47.958 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-10T03:53:47.970 INFO:teuthology.orchestra.run.smithi006.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 10 03:53 /home/ubuntu/cephtest 2024-03-10T03:53:47.972 INFO:teuthology.orchestra.run.smithi071.stdout: 265842 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 10 03:53 /home/ubuntu/cephtest 2024-03-10T03:53:47.973 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-10T03:53:47.989 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-10T03:53:48.082 INFO:teuthology.nuke:Checking targets against current locks 2024-03-10T03:53:48.109 DEBUG:teuthology.nuke:shortname: smithi006 2024-03-10T03:53:48.109 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-10T03:53:48.129 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587949', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-10 03:24:59.728999', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD1F9RkJkfkAKjjgkfEOQ0i3yUMmXuwZ5bqI4CJd1lFmbJo66FSXFjQEGM+LbgP3CNluxm9nXM3YpBtF/CGB85E='} 2024-03-10T03:53:48.134 DEBUG:teuthology.nuke:shortname: smithi071 2024-03-10T03:53:48.134 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-10T03:53:48.156 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-08_16:20:46-rados-wip-yuri4-testing-2024-03-05-0854-distro-default-smithi/7587949', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-10 03:24:59.727002', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAbrs4181qoZ3+51Gog2JhqrEsleVQk5LCcisz7vdQfkUqS16qOD7CFmreJVG0MJkw5/JCOo02pDWdBm3QnHygU='} 2024-03-10T03:53:48.171 INFO:teuthology.orchestra.console.smithi006:Power off 2024-03-10T03:53:48.171 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-10T03:53:48.197 DEBUG:teuthology.orchestra.console.smithi006:power off output: Chassis Power Control: Down/Off 2024-03-10T03:53:48.197 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-10T03:53:48.213 INFO:teuthology.orchestra.console.smithi071:Power off 2024-03-10T03:53:48.213 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-10T03:53:48.223 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2024-03-10T03:53:48.236 DEBUG:teuthology.orchestra.console.smithi071:power off output: Chassis Power Control: Down/Off 2024-03-10T03:53:48.236 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-10T03:53:48.258 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on 2024-03-10T03:53:52.224 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-10T03:53:52.259 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-10T03:53:52.333 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2024-03-10T03:53:52.368 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on 2024-03-10T03:53:56.338 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-10T03:53:56.369 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-10T03:53:56.452 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is off 2024-03-10T03:53:56.480 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on 2024-03-10T03:53:56.553 INFO:teuthology.orchestra.console.smithi006:Power off completed 2024-03-10T03:54:00.482 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-10T03:54:00.592 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is off 2024-03-10T03:54:00.693 INFO:teuthology.orchestra.console.smithi071:Power off completed 2024-03-10T03:54:00.796 INFO:teuthology.run:Summary data: description: rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} duration: 919.4651308059692 failure_reason: '"2024-03-10T03:45:23.518476+0000 mon.a (mon.0) 212 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b" in cluster log' owner: scheduled_yuriw@teuthology success: false 2024-03-10T03:54:00.797 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-10T03:54:00.897 INFO:teuthology.run:FAIL