2026-02-14 01:33:29,633 7f0b59619e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'pull'] 2026-02-14 01:33:29,687 7f0b59619e00 INFO Pulling container image quay.ceph.io/ceph-ci/ceph:squid... 2026-02-14 01:33:37,732 7f0b59619e00 DEBUG /bin/podman: stdout 4cc07a41bb1966e9ffd51d6ffba181735181325200aceb7b3d1b7e810c433126,[quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4] 2026-02-14 01:33:37,862 7f0b59619e00 DEBUG ceph: stdout ceph version 19.2.3-672-g283431c0 (283431c091dafc302f5753f809d0bd1eddce0869) squid (stable) 2026-02-14 01:33:38,376 7f9b6568ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', '-v', 'bootstrap', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '10.20.193.14', '--skip-admin-label'] 2026-02-14 01:33:38,377 7f9b6568ce00 WARNING Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2026-02-14 01:33:38,377 7f9b6568ce00 INFO Verifying podman|docker is present... 2026-02-14 01:33:38,393 7f9b6568ce00 QUIET /bin/podman: stdout 5.6.0 2026-02-14 01:33:38,394 7f9b6568ce00 INFO Verifying lvm2 is present... 2026-02-14 01:33:38,394 7f9b6568ce00 INFO Verifying time synchronization is in place... 2026-02-14 01:33:38,400 7f9b6568ce00 QUIET Non-zero exit code 1 from systemctl is-enabled chrony.service 2026-02-14 01:33:38,400 7f9b6568ce00 QUIET systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2026-02-14 01:33:38,407 7f9b6568ce00 QUIET Non-zero exit code 3 from systemctl is-active chrony.service 2026-02-14 01:33:38,407 7f9b6568ce00 QUIET systemctl: stdout inactive 2026-02-14 01:33:38,413 7f9b6568ce00 QUIET systemctl: stdout enabled 2026-02-14 01:33:38,419 7f9b6568ce00 QUIET systemctl: stdout active 2026-02-14 01:33:38,419 7f9b6568ce00 INFO Unit chronyd.service is enabled and running 2026-02-14 01:33:38,419 7f9b6568ce00 INFO Repeating the final host check... 2026-02-14 01:33:38,434 7f9b6568ce00 QUIET /bin/podman: stdout 5.6.0 2026-02-14 01:33:38,434 7f9b6568ce00 INFO podman (/bin/podman) version 5.6.0 is present 2026-02-14 01:33:38,434 7f9b6568ce00 INFO systemctl is present 2026-02-14 01:33:38,434 7f9b6568ce00 INFO lvcreate is present 2026-02-14 01:33:38,439 7f9b6568ce00 QUIET Non-zero exit code 1 from systemctl is-enabled chrony.service 2026-02-14 01:33:38,439 7f9b6568ce00 QUIET systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2026-02-14 01:33:38,445 7f9b6568ce00 QUIET Non-zero exit code 3 from systemctl is-active chrony.service 2026-02-14 01:33:38,445 7f9b6568ce00 QUIET systemctl: stdout inactive 2026-02-14 01:33:38,452 7f9b6568ce00 QUIET systemctl: stdout enabled 2026-02-14 01:33:38,457 7f9b6568ce00 QUIET systemctl: stdout active 2026-02-14 01:33:38,457 7f9b6568ce00 INFO Unit chronyd.service is enabled and running 2026-02-14 01:33:38,457 7f9b6568ce00 INFO Host looks OK 2026-02-14 01:33:38,457 7f9b6568ce00 INFO Cluster fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:33:38,458 7f9b6568ce00 QUIET Acquiring lock 140305370820768 on /run/cephadm/29ce0a10-0945-11f1-b2be-d404e6e7d460.lock 2026-02-14 01:33:38,458 7f9b6568ce00 QUIET Lock 140305370820768 acquired on /run/cephadm/29ce0a10-0945-11f1-b2be-d404e6e7d460.lock 2026-02-14 01:33:38,458 7f9b6568ce00 INFO Verifying IP 10.20.193.14 port 3300 ... 2026-02-14 01:33:38,458 7f9b6568ce00 INFO Verifying IP 10.20.193.14 port 6789 ... 2026-02-14 01:33:38,458 7f9b6568ce00 DEBUG Base mon IP(s) is [10.20.193.14:3300, 10.20.193.14:6789], mon addrv is [v2:10.20.193.14:3300,v1:10.20.193.14:6789] 2026-02-14 01:33:38,461 7f9b6568ce00 QUIET /sbin/ip: stdout default via 10.20.192.1 dev enp1s0f0 proto dhcp src 10.20.193.14 metric 102 2026-02-14 01:33:38,462 7f9b6568ce00 QUIET /sbin/ip: stdout 10.20.192.0/20 dev enp1s0f0 proto kernel scope link src 10.20.193.14 metric 102 2026-02-14 01:33:38,462 7f9b6568ce00 QUIET /sbin/ip: stdout 169.254.3.0/24 dev enp7s0f4u2u2c2 proto kernel scope link src 169.254.3.1 metric 101 2026-02-14 01:33:38,464 7f9b6568ce00 QUIET /sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2026-02-14 01:33:38,464 7f9b6568ce00 QUIET /sbin/ip: stdout fe80::/64 dev enp1s0f0 proto kernel metric 256 pref medium 2026-02-14 01:33:38,464 7f9b6568ce00 QUIET /sbin/ip: stdout fe80::/64 dev enp7s0f4u2u2c2 proto kernel metric 1024 pref medium 2026-02-14 01:33:38,466 7f9b6568ce00 QUIET /sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2026-02-14 01:33:38,467 7f9b6568ce00 QUIET /sbin/ip: stdout inet6 ::1/128 scope host 2026-02-14 01:33:38,467 7f9b6568ce00 QUIET /sbin/ip: stdout valid_lft forever preferred_lft forever 2026-02-14 01:33:38,467 7f9b6568ce00 QUIET /sbin/ip: stdout 2: enp7s0f4u2u2c2: mtu 1500 state UNKNOWN qlen 1000 2026-02-14 01:33:38,467 7f9b6568ce00 QUIET /sbin/ip: stdout inet6 fe80::9730:4b5a:f3d2:a008/64 scope link noprefixroute 2026-02-14 01:33:38,467 7f9b6568ce00 QUIET /sbin/ip: stdout valid_lft forever preferred_lft forever 2026-02-14 01:33:38,467 7f9b6568ce00 QUIET /sbin/ip: stdout 3: enp1s0f0: mtu 1500 state UP qlen 1000 2026-02-14 01:33:38,467 7f9b6568ce00 QUIET /sbin/ip: stdout inet6 fe80::925a:8ff:fe77:6034/64 scope link 2026-02-14 01:33:38,467 7f9b6568ce00 QUIET /sbin/ip: stdout valid_lft forever preferred_lft forever 2026-02-14 01:33:38,467 7f9b6568ce00 INFO Mon IP `10.20.193.14` is in CIDR network `10.20.192.0/20` 2026-02-14 01:33:38,467 7f9b6568ce00 INFO Mon IP `10.20.193.14` is in CIDR network `10.20.192.0/20` 2026-02-14 01:33:38,467 7f9b6568ce00 DEBUG Inferred mon public CIDR from local network configuration ['10.20.192.0/20', '10.20.192.0/20'] 2026-02-14 01:33:38,467 7f9b6568ce00 INFO Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2026-02-14 01:33:38,468 7f9b6568ce00 INFO Pulling container image quay.ceph.io/ceph-ci/ceph:squid... 2026-02-14 01:33:38,711 7f9b6568ce00 QUIET /bin/podman: stdout 4cc07a41bb1966e9ffd51d6ffba181735181325200aceb7b3d1b7e810c433126 2026-02-14 01:33:38,711 7f9b6568ce00 QUIET /bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:squid... 2026-02-14 01:33:38,711 7f9b6568ce00 QUIET /bin/podman: stderr Getting image source signatures 2026-02-14 01:33:38,711 7f9b6568ce00 QUIET /bin/podman: stderr Copying blob sha256:c92ccfe8c151fb35c12ff15a7a7de08028e120d73814d436eadc192e84f49ad9 2026-02-14 01:33:38,711 7f9b6568ce00 QUIET /bin/podman: stderr Copying blob sha256:64e40020cdc81486fe697dfe0e523a5804d33e01a9e9bdfc58898ab18d415677 2026-02-14 01:33:38,711 7f9b6568ce00 QUIET /bin/podman: stderr Copying config sha256:4cc07a41bb1966e9ffd51d6ffba181735181325200aceb7b3d1b7e810c433126 2026-02-14 01:33:38,711 7f9b6568ce00 QUIET /bin/podman: stderr Writing manifest to image destination 2026-02-14 01:33:38,835 7f9b6568ce00 DEBUG ceph: stdout ceph version 19.2.3-672-g283431c0 (283431c091dafc302f5753f809d0bd1eddce0869) squid (stable) 2026-02-14 01:33:38,836 7f9b6568ce00 INFO Ceph version: ceph version 19.2.3-672-g283431c0 (283431c091dafc302f5753f809d0bd1eddce0869) squid (stable) 2026-02-14 01:33:38,836 7f9b6568ce00 INFO Extracting ceph user uid/gid from container image... 2026-02-14 01:33:38,977 7f9b6568ce00 QUIET stat: stdout 167 167 2026-02-14 01:33:38,978 7f9b6568ce00 INFO Creating initial keys... 2026-02-14 01:33:39,125 7f9b6568ce00 DEBUG /usr/bin/ceph-authtool: stdout AQDz0I9p5ZHrBRAAqsLJRevJFlCa8pfULLdJ7Q== 2026-02-14 01:33:39,269 7f9b6568ce00 DEBUG /usr/bin/ceph-authtool: stdout AQDz0I9pq+/HDhAAAFmWXX1EN3RlEjUx8y63ig== 2026-02-14 01:33:39,396 7f9b6568ce00 DEBUG /usr/bin/ceph-authtool: stdout AQDz0I9pYGlEFhAAOnF6ryAzuu/H9SeCFjrELQ== 2026-02-14 01:33:39,397 7f9b6568ce00 INFO Creating initial monmap... 2026-02-14 01:33:39,520 7f9b6568ce00 DEBUG /usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2026-02-14 01:33:39,520 7f9b6568ce00 DEBUG /usr/bin/monmaptool: stdout setting min_mon_release = quincy 2026-02-14 01:33:39,520 7f9b6568ce00 DEBUG /usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:33:39,520 7f9b6568ce00 DEBUG /usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2026-02-14 01:33:39,520 7f9b6568ce00 DEBUG monmaptool for trial014 [v2:10.20.193.14:3300,v1:10.20.193.14:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap setting min_mon_release = quincy /usr/bin/monmaptool: set fsid to 29ce0a10-0945-11f1-b2be-d404e6e7d460 /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2026-02-14 01:33:39,520 7f9b6568ce00 INFO Creating mon... 2026-02-14 01:33:39,650 7f9b6568ce00 DEBUG create mon.trial014 on 2026-02-14 01:33:39,833 7f9b6568ce00 DEBUG systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2026-02-14 01:33:39,938 7f9b6568ce00 DEBUG systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2026-02-14 01:33:40,043 7f9b6568ce00 DEBUG systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460.target → /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460.target. 2026-02-14 01:33:40,043 7f9b6568ce00 DEBUG systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460.target → /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460.target. 2026-02-14 01:33:40,165 7f9b6568ce00 DEBUG Non-zero exit code 1 from systemctl reset-failed ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@mon.trial014 2026-02-14 01:33:40,166 7f9b6568ce00 DEBUG systemctl: stderr Failed to reset failed state of unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@mon.trial014.service: Unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@mon.trial014.service not loaded. 2026-02-14 01:33:40,281 7f9b6568ce00 DEBUG systemctl: stderr Created symlink /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460.target.wants/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@mon.trial014.service → /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@.service. 2026-02-14 01:33:40,435 7f9b6568ce00 QUIET Non-zero exit code 1 from systemctl is-enabled firewalld.service 2026-02-14 01:33:40,435 7f9b6568ce00 QUIET systemctl: stdout disabled 2026-02-14 01:33:40,442 7f9b6568ce00 QUIET Non-zero exit code 3 from systemctl is-active firewalld.service 2026-02-14 01:33:40,442 7f9b6568ce00 QUIET systemctl: stdout inactive 2026-02-14 01:33:40,442 7f9b6568ce00 DEBUG firewalld.service is not enabled 2026-02-14 01:33:40,442 7f9b6568ce00 DEBUG Not possible to enable service . firewalld.service is not available 2026-02-14 01:33:40,442 7f9b6568ce00 INFO Waiting for mon to start... 2026-02-14 01:33:40,442 7f9b6568ce00 INFO Waiting for mon... 2026-02-14 01:33:40,608 7f9b6568ce00 QUIET /usr/bin/ceph: stdout cluster: 2026-02-14 01:33:40,608 7f9b6568ce00 QUIET /usr/bin/ceph: stdout id: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:33:40,608 7f9b6568ce00 QUIET /usr/bin/ceph: stdout health: HEALTH_OK 2026-02-14 01:33:40,608 7f9b6568ce00 QUIET /usr/bin/ceph: stdout 2026-02-14 01:33:40,608 7f9b6568ce00 QUIET /usr/bin/ceph: stdout services: 2026-02-14 01:33:40,608 7f9b6568ce00 QUIET /usr/bin/ceph: stdout mon: 1 daemons, quorum trial014 (age 0.143709s) 2026-02-14 01:33:40,608 7f9b6568ce00 QUIET /usr/bin/ceph: stdout mgr: no daemons active 2026-02-14 01:33:40,608 7f9b6568ce00 QUIET /usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2026-02-14 01:33:40,608 7f9b6568ce00 QUIET /usr/bin/ceph: stdout 2026-02-14 01:33:40,608 7f9b6568ce00 QUIET /usr/bin/ceph: stdout data: 2026-02-14 01:33:40,608 7f9b6568ce00 QUIET /usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2026-02-14 01:33:40,608 7f9b6568ce00 QUIET /usr/bin/ceph: stdout objects: 0 objects, 0 B 2026-02-14 01:33:40,609 7f9b6568ce00 QUIET /usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2026-02-14 01:33:40,609 7f9b6568ce00 QUIET /usr/bin/ceph: stdout pgs: 2026-02-14 01:33:40,609 7f9b6568ce00 QUIET /usr/bin/ceph: stdout 2026-02-14 01:33:40,609 7f9b6568ce00 INFO mon is available 2026-02-14 01:33:40,609 7f9b6568ce00 INFO Assimilating anything we can from ceph.conf... 2026-02-14 01:33:40,750 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout 2026-02-14 01:33:40,751 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout [global] 2026-02-14 01:33:40,751 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout fsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:33:40,751 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2026-02-14 01:33:40,751 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout mon_host = [v2:10.20.193.14:3300,v1:10.20.193.14:6789] 2026-02-14 01:33:40,751 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2026-02-14 01:33:40,751 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2026-02-14 01:33:40,751 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2026-02-14 01:33:40,751 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2026-02-14 01:33:40,751 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout 2026-02-14 01:33:40,751 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout [mgr] 2026-02-14 01:33:40,751 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout mgr/telemetry/nag = false 2026-02-14 01:33:40,751 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout 2026-02-14 01:33:40,751 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout [osd] 2026-02-14 01:33:40,751 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2026-02-14 01:33:40,751 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2026-02-14 01:33:40,751 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2026-02-14 01:33:40,751 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout osd_map_max_advance = 10 2026-02-14 01:33:40,751 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout osd_sloppy_crc = true 2026-02-14 01:33:40,751 7f9b6568ce00 INFO Generating new minimal ceph.conf... 2026-02-14 01:33:40,907 7f9b6568ce00 INFO Restarting the monitor... 2026-02-14 01:33:41,414 7f9b6568ce00 INFO Setting public_network to 10.20.192.0/20 in mon config section 2026-02-14 01:33:41,568 7f9b6568ce00 INFO Wrote config to /etc/ceph/ceph.conf 2026-02-14 01:33:41,568 7f9b6568ce00 INFO Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2026-02-14 01:33:41,568 7f9b6568ce00 INFO Creating mgr... 2026-02-14 01:33:41,569 7f9b6568ce00 INFO Verifying port 0.0.0.0:9283 ... 2026-02-14 01:33:41,569 7f9b6568ce00 INFO Verifying port 0.0.0.0:8765 ... 2026-02-14 01:33:41,569 7f9b6568ce00 INFO Verifying port 0.0.0.0:8443 ... 2026-02-14 01:33:41,693 7f9b6568ce00 DEBUG Non-zero exit code 1 from systemctl reset-failed ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@mgr.trial014.uucrmp 2026-02-14 01:33:41,693 7f9b6568ce00 DEBUG systemctl: stderr Failed to reset failed state of unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@mgr.trial014.uucrmp.service: Unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@mgr.trial014.uucrmp.service not loaded. 2026-02-14 01:33:41,807 7f9b6568ce00 DEBUG systemctl: stderr Created symlink /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460.target.wants/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@mgr.trial014.uucrmp.service → /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@.service. 2026-02-14 01:33:41,942 7f9b6568ce00 QUIET Non-zero exit code 1 from systemctl is-enabled firewalld.service 2026-02-14 01:33:41,942 7f9b6568ce00 QUIET systemctl: stdout disabled 2026-02-14 01:33:41,948 7f9b6568ce00 QUIET Non-zero exit code 3 from systemctl is-active firewalld.service 2026-02-14 01:33:41,948 7f9b6568ce00 QUIET systemctl: stdout inactive 2026-02-14 01:33:41,948 7f9b6568ce00 DEBUG firewalld.service is not enabled 2026-02-14 01:33:41,949 7f9b6568ce00 DEBUG Not possible to enable service . firewalld.service is not available 2026-02-14 01:33:41,956 7f9b6568ce00 QUIET Non-zero exit code 1 from systemctl is-enabled firewalld.service 2026-02-14 01:33:41,956 7f9b6568ce00 QUIET systemctl: stdout disabled 2026-02-14 01:33:41,961 7f9b6568ce00 QUIET Non-zero exit code 3 from systemctl is-active firewalld.service 2026-02-14 01:33:41,962 7f9b6568ce00 QUIET systemctl: stdout inactive 2026-02-14 01:33:41,962 7f9b6568ce00 DEBUG firewalld.service is not enabled 2026-02-14 01:33:41,962 7f9b6568ce00 DEBUG Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2026-02-14 01:33:41,962 7f9b6568ce00 INFO Waiting for mgr to start... 2026-02-14 01:33:41,962 7f9b6568ce00 INFO Waiting for mgr... 2026-02-14 01:33:42,130 7f9b6568ce00 QUIET /usr/bin/ceph: stdout 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout { 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "fsid": "29ce0a10-0945-11f1-b2be-d404e6e7d460", 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "health": { 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "status": "HEALTH_OK", 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "checks": {}, 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "mutes": [] 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout }, 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "election_epoch": 5, 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "quorum": [ 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout 0 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout ], 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "quorum_names": [ 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "trial014" 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout ], 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "quorum_age": 0, 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "monmap": { 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "epoch": 1, 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "min_mon_release_name": "squid", 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_mons": 1 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout }, 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "osdmap": { 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "epoch": 1, 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_osds": 0, 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_up_osds": 0, 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "osd_up_since": 0, 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_in_osds": 0, 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "osd_in_since": 0, 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_remapped_pgs": 0 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout }, 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "pgmap": { 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "pgs_by_state": [], 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_pgs": 0, 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_pools": 0, 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_objects": 0, 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "data_bytes": 0, 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "bytes_used": 0, 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "bytes_avail": 0, 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "bytes_total": 0 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout }, 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "fsmap": { 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "epoch": 1, 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "btime": "2026-02-14T01:33:40:447598+0000", 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "by_rank": [], 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "up:standby": 0 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout }, 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "mgrmap": { 2026-02-14 01:33:42,131 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "available": false, 2026-02-14 01:33:42,132 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_standbys": 0, 2026-02-14 01:33:42,132 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "modules": [ 2026-02-14 01:33:42,132 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "iostat", 2026-02-14 01:33:42,132 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "nfs", 2026-02-14 01:33:42,132 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "restful" 2026-02-14 01:33:42,132 7f9b6568ce00 QUIET /usr/bin/ceph: stdout ], 2026-02-14 01:33:42,132 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "services": {} 2026-02-14 01:33:42,132 7f9b6568ce00 QUIET /usr/bin/ceph: stdout }, 2026-02-14 01:33:42,132 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "servicemap": { 2026-02-14 01:33:42,132 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "epoch": 1, 2026-02-14 01:33:42,132 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "modified": "2026-02-14T01:33:40.448087+0000", 2026-02-14 01:33:42,132 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "services": {} 2026-02-14 01:33:42,132 7f9b6568ce00 QUIET /usr/bin/ceph: stdout }, 2026-02-14 01:33:42,132 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "progress_events": {} 2026-02-14 01:33:42,132 7f9b6568ce00 QUIET /usr/bin/ceph: stdout } 2026-02-14 01:33:42,132 7f9b6568ce00 INFO mgr not available, waiting (1/15)... 2026-02-14 01:33:44,284 7f9b6568ce00 QUIET /usr/bin/ceph: stdout 2026-02-14 01:33:44,284 7f9b6568ce00 QUIET /usr/bin/ceph: stdout { 2026-02-14 01:33:44,284 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "fsid": "29ce0a10-0945-11f1-b2be-d404e6e7d460", 2026-02-14 01:33:44,284 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "health": { 2026-02-14 01:33:44,284 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "status": "HEALTH_OK", 2026-02-14 01:33:44,284 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "checks": {}, 2026-02-14 01:33:44,284 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "mutes": [] 2026-02-14 01:33:44,284 7f9b6568ce00 QUIET /usr/bin/ceph: stdout }, 2026-02-14 01:33:44,284 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "election_epoch": 5, 2026-02-14 01:33:44,284 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "quorum": [ 2026-02-14 01:33:44,284 7f9b6568ce00 QUIET /usr/bin/ceph: stdout 0 2026-02-14 01:33:44,284 7f9b6568ce00 QUIET /usr/bin/ceph: stdout ], 2026-02-14 01:33:44,284 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "quorum_names": [ 2026-02-14 01:33:44,284 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "trial014" 2026-02-14 01:33:44,284 7f9b6568ce00 QUIET /usr/bin/ceph: stdout ], 2026-02-14 01:33:44,284 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "quorum_age": 2, 2026-02-14 01:33:44,284 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "monmap": { 2026-02-14 01:33:44,284 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "epoch": 1, 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "min_mon_release_name": "squid", 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_mons": 1 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout }, 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "osdmap": { 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "epoch": 1, 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_osds": 0, 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_up_osds": 0, 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "osd_up_since": 0, 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_in_osds": 0, 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "osd_in_since": 0, 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_remapped_pgs": 0 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout }, 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "pgmap": { 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "pgs_by_state": [], 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_pgs": 0, 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_pools": 0, 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_objects": 0, 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "data_bytes": 0, 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "bytes_used": 0, 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "bytes_avail": 0, 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "bytes_total": 0 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout }, 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "fsmap": { 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "epoch": 1, 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "btime": "2026-02-14T01:33:40:447598+0000", 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "by_rank": [], 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "up:standby": 0 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout }, 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "mgrmap": { 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "available": false, 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_standbys": 0, 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "modules": [ 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "iostat", 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "nfs", 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "restful" 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout ], 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "services": {} 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout }, 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "servicemap": { 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "epoch": 1, 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "modified": "2026-02-14T01:33:40.448087+0000", 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "services": {} 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout }, 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "progress_events": {} 2026-02-14 01:33:44,285 7f9b6568ce00 QUIET /usr/bin/ceph: stdout } 2026-02-14 01:33:44,285 7f9b6568ce00 INFO mgr not available, waiting (2/15)... 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout { 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "fsid": "29ce0a10-0945-11f1-b2be-d404e6e7d460", 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "health": { 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "status": "HEALTH_OK", 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "checks": {}, 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "mutes": [] 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout }, 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "election_epoch": 5, 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "quorum": [ 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout 0 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout ], 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "quorum_names": [ 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "trial014" 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout ], 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "quorum_age": 5, 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "monmap": { 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "epoch": 1, 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "min_mon_release_name": "squid", 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_mons": 1 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout }, 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "osdmap": { 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "epoch": 1, 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_osds": 0, 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_up_osds": 0, 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "osd_up_since": 0, 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_in_osds": 0, 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "osd_in_since": 0, 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_remapped_pgs": 0 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout }, 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "pgmap": { 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "pgs_by_state": [], 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_pgs": 0, 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_pools": 0, 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_objects": 0, 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "data_bytes": 0, 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "bytes_used": 0, 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "bytes_avail": 0, 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "bytes_total": 0 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout }, 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "fsmap": { 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "epoch": 1, 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "btime": "2026-02-14T01:33:40:447598+0000", 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "by_rank": [], 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "up:standby": 0 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout }, 2026-02-14 01:33:46,536 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "mgrmap": { 2026-02-14 01:33:46,537 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "available": true, 2026-02-14 01:33:46,537 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_standbys": 0, 2026-02-14 01:33:46,537 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "modules": [ 2026-02-14 01:33:46,537 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "iostat", 2026-02-14 01:33:46,537 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "nfs", 2026-02-14 01:33:46,537 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "restful" 2026-02-14 01:33:46,537 7f9b6568ce00 QUIET /usr/bin/ceph: stdout ], 2026-02-14 01:33:46,537 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "services": {} 2026-02-14 01:33:46,537 7f9b6568ce00 QUIET /usr/bin/ceph: stdout }, 2026-02-14 01:33:46,537 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "servicemap": { 2026-02-14 01:33:46,537 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "epoch": 1, 2026-02-14 01:33:46,537 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "modified": "2026-02-14T01:33:40.448087+0000", 2026-02-14 01:33:46,537 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "services": {} 2026-02-14 01:33:46,537 7f9b6568ce00 QUIET /usr/bin/ceph: stdout }, 2026-02-14 01:33:46,537 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "progress_events": {} 2026-02-14 01:33:46,537 7f9b6568ce00 QUIET /usr/bin/ceph: stdout } 2026-02-14 01:33:46,537 7f9b6568ce00 INFO mgr is available 2026-02-14 01:33:46,742 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout 2026-02-14 01:33:46,742 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout [global] 2026-02-14 01:33:46,742 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout fsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:33:46,742 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2026-02-14 01:33:46,742 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout mon_host = [v2:10.20.193.14:3300,v1:10.20.193.14:6789] 2026-02-14 01:33:46,743 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2026-02-14 01:33:46,743 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2026-02-14 01:33:46,743 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2026-02-14 01:33:46,743 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2026-02-14 01:33:46,743 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout 2026-02-14 01:33:46,743 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout [mgr] 2026-02-14 01:33:46,743 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout mgr/telemetry/nag = false 2026-02-14 01:33:46,743 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout 2026-02-14 01:33:46,743 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout [osd] 2026-02-14 01:33:46,743 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2026-02-14 01:33:46,743 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2026-02-14 01:33:46,743 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2026-02-14 01:33:46,743 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout osd_map_max_advance = 10 2026-02-14 01:33:46,743 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout osd_sloppy_crc = true 2026-02-14 01:33:46,743 7f9b6568ce00 INFO Enabling cephadm module... 2026-02-14 01:33:47,858 7f9b6568ce00 QUIET /usr/bin/ceph: stdout { 2026-02-14 01:33:47,859 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "epoch": 5, 2026-02-14 01:33:47,859 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "available": true, 2026-02-14 01:33:47,859 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "active_name": "trial014.uucrmp", 2026-02-14 01:33:47,859 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_standby": 0 2026-02-14 01:33:47,859 7f9b6568ce00 QUIET /usr/bin/ceph: stdout } 2026-02-14 01:33:47,859 7f9b6568ce00 INFO Waiting for the mgr to restart... 2026-02-14 01:33:47,859 7f9b6568ce00 INFO Waiting for mgr epoch 5... 2026-02-14 01:33:51,292 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout { 2026-02-14 01:33:51,292 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout "mgrmap_epoch": 7, 2026-02-14 01:33:51,292 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout "initialized": true 2026-02-14 01:33:51,292 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout } 2026-02-14 01:33:51,292 7f9b6568ce00 INFO mgr epoch 5 is available 2026-02-14 01:33:51,292 7f9b6568ce00 INFO Setting orchestrator backend to cephadm... 2026-02-14 01:33:51,723 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout value unchanged 2026-02-14 01:33:51,723 7f9b6568ce00 INFO Generating ssh key... 2026-02-14 01:33:52,141 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILAIdDCvTqiaI1u5w2jsomXn2ds236BU6a+h8kdnWQwb ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:33:52,142 7f9b6568ce00 INFO Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2026-02-14 01:33:52,142 7f9b6568ce00 INFO Adding key to root@localhost authorized_keys... 2026-02-14 01:33:52,143 7f9b6568ce00 INFO Adding host trial014... 2026-02-14 01:33:53,730 7f0d290cae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host', '--expect-hostname', 'trial014'] 2026-02-14 01:33:53,748 7f0d290cae00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 01:33:53,748 7f0d290cae00 INFO systemctl is present 2026-02-14 01:33:53,748 7f0d290cae00 INFO lvcreate is present 2026-02-14 01:33:53,772 7f0d290cae00 INFO Unit chronyd.service is enabled and running 2026-02-14 01:33:53,772 7f0d290cae00 INFO Hostname "trial014" matches what is expected. 2026-02-14 01:33:53,772 7f0d290cae00 INFO Host looks OK 2026-02-14 01:33:53,813 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout Added host 'trial014' with addr '10.20.193.14' 2026-02-14 01:33:53,813 7f9b6568ce00 INFO Deploying mon service with default placement... 2026-02-14 01:33:53,895 7f9c7d49ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', '--timeout', '895', 'pull'] 2026-02-14 01:33:53,911 7f9c7d49ae00 INFO Pulling container image quay.ceph.io/ceph-ci/ceph:squid... 2026-02-14 01:33:54,042 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout Scheduled mon update... 2026-02-14 01:33:54,042 7f9b6568ce00 INFO Deploying mgr service with default placement... 2026-02-14 01:33:54,176 7f9c7d49ae00 DEBUG /usr/bin/podman: stdout 4cc07a41bb1966e9ffd51d6ffba181735181325200aceb7b3d1b7e810c433126,[quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4] 2026-02-14 01:33:54,264 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout Scheduled mgr update... 2026-02-14 01:33:54,264 7f9b6568ce00 INFO Deploying crash service with default placement... 2026-02-14 01:33:54,299 7f9c7d49ae00 DEBUG ceph: stdout ceph version 19.2.3-672-g283431c0 (283431c091dafc302f5753f809d0bd1eddce0869) squid (stable) 2026-02-14 01:33:54,427 7f0802c7ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 01:33:54,443 7f0802c7ce00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 01:33:54,443 7f0802c7ce00 INFO systemctl is present 2026-02-14 01:33:54,443 7f0802c7ce00 INFO lvcreate is present 2026-02-14 01:33:54,467 7f0802c7ce00 INFO Unit chronyd.service is enabled and running 2026-02-14 01:33:54,467 7f0802c7ce00 INFO Host looks OK 2026-02-14 01:33:54,490 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout Scheduled crash update... 2026-02-14 01:33:54,490 7f9b6568ce00 INFO Deploying ceph-exporter service with default placement... 2026-02-14 01:33:54,591 7f2269610e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ls'] 2026-02-14 01:33:54,716 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout Scheduled ceph-exporter update... 2026-02-14 01:33:54,716 7f9b6568ce00 INFO Deploying prometheus service with default placement... 2026-02-14 01:33:54,931 7fb1624dae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:33:54,939 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout Scheduled prometheus update... 2026-02-14 01:33:54,939 7f9b6568ce00 INFO Deploying grafana service with default placement... 2026-02-14 01:33:55,149 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout Scheduled grafana update... 2026-02-14 01:33:55,150 7f9b6568ce00 INFO Deploying node-exporter service with default placement... 2026-02-14 01:33:55,170 7f3eddbe1e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'list-networks'] 2026-02-14 01:33:55,313 7fcbc00a6e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 01:33:55,370 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout Scheduled node-exporter update... 2026-02-14 01:33:55,370 7f9b6568ce00 INFO Deploying alertmanager service with default placement... 2026-02-14 01:33:55,377 7fcbc00a6e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:33:55,377 7fcbc00a6e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:33:55,590 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout Scheduled alertmanager update... 2026-02-14 01:33:55,999 7f9b6568ce00 INFO Enabling the dashboard module... 2026-02-14 01:33:57,249 7f9b6568ce00 QUIET /usr/bin/ceph: stdout { 2026-02-14 01:33:57,249 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "epoch": 9, 2026-02-14 01:33:57,249 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "available": true, 2026-02-14 01:33:57,249 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "active_name": "trial014.uucrmp", 2026-02-14 01:33:57,249 7f9b6568ce00 QUIET /usr/bin/ceph: stdout "num_standby": 0 2026-02-14 01:33:57,249 7f9b6568ce00 QUIET /usr/bin/ceph: stdout } 2026-02-14 01:33:57,249 7f9b6568ce00 INFO Waiting for the mgr to restart... 2026-02-14 01:33:57,249 7f9b6568ce00 INFO Waiting for mgr epoch 9... 2026-02-14 01:33:59,892 7f75682d5e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ls'] 2026-02-14 01:34:00,232 7fc6ec243e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:34:00,445 7f5cb6046e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'list-networks'] 2026-02-14 01:34:00,587 7fa718a3ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 01:34:00,639 7fa718a3ce00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:00,639 7fa718a3ce00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:00,695 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout { 2026-02-14 01:34:00,695 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout "mgrmap_epoch": 11, 2026-02-14 01:34:00,695 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout "initialized": true 2026-02-14 01:34:00,695 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout } 2026-02-14 01:34:00,695 7f9b6568ce00 INFO mgr epoch 9 is available 2026-02-14 01:34:00,695 7f9b6568ce00 INFO Generating a dashboard self-signed certificate... 2026-02-14 01:34:00,965 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout Self-signed certificate created 2026-02-14 01:34:00,965 7f9b6568ce00 INFO Creating initial admin user... 2026-02-14 01:34:01,317 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$/.mRxAhGhTSPkBuJrB37d.uKxz7R1etmtgcYlNGYwDiuJLjWPgml2", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1771032841, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2026-02-14 01:34:01,317 7f9b6568ce00 INFO Fetching dashboard port number... 2026-02-14 01:34:01,523 7f9b6568ce00 DEBUG /usr/bin/ceph: stdout 8443 2026-02-14 01:34:01,532 7f9b6568ce00 QUIET Non-zero exit code 1 from systemctl is-enabled firewalld.service 2026-02-14 01:34:01,532 7f9b6568ce00 QUIET systemctl: stdout disabled 2026-02-14 01:34:01,538 7f9b6568ce00 QUIET Non-zero exit code 3 from systemctl is-active firewalld.service 2026-02-14 01:34:01,538 7f9b6568ce00 QUIET systemctl: stdout inactive 2026-02-14 01:34:01,538 7f9b6568ce00 DEBUG firewalld.service is not enabled 2026-02-14 01:34:01,538 7f9b6568ce00 DEBUG Not possible to open ports <[8443]>. firewalld.service is not available 2026-02-14 01:34:01,539 7f9b6568ce00 INFO Ceph Dashboard is now available at: URL: https://trial014:8443/ User: admin Password: 671aendt18 2026-02-14 01:34:01,540 7f9b6568ce00 INFO Saving cluster configuration to /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config directory 2026-02-14 01:34:01,788 7f9b6568ce00 DEBUG /usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2026-02-14 01:34:01,788 7f9b6568ce00 INFO You can access the Ceph CLI as following in case of multi-cluster or non-default config: sudo /home/ubuntu/cephtest/cephadm shell --fsid 29ce0a10-0945-11f1-b2be-d404e6e7d460 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2026-02-14 01:34:01,788 7f9b6568ce00 INFO Or, if you are only running a single cluster on this host: sudo /home/ubuntu/cephtest/cephadm shell 2026-02-14 01:34:01,788 7f9b6568ce00 INFO Please consider enabling telemetry to help improve Ceph: ceph telemetry on For more information see: https://docs.ceph.com/en/latest/mgr/telemetry/ 2026-02-14 01:34:01,788 7f9b6568ce00 INFO Bootstrap complete. 2026-02-14 01:34:02,405 7fab582d6e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/cephadm/allow_ptrace', 'true'] 2026-02-14 01:34:02,790 7fab582d6e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:02,790 7fab582d6e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:02,790 7fab582d6e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:02,790 7fab582d6e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config set mgr mgr/cephadm/allow_ptrace true 2026-02-14 01:34:03,259 7f80123d2e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'orch', 'client-keyring', 'set', 'client.admin', '*', '--mode', '0755'] 2026-02-14 01:34:03,314 7f80123d2e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:03,314 7f80123d2e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:03,314 7f80123d2e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:03,315 7f80123d2e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid orch client-keyring set client.admin * --mode 0755 2026-02-14 01:34:03,392 7efe3dbbce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:34:03,408 7efe3dbbce00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'ceph-exporter.trial014', 'image': '', 'deploy_arguments': [], 'params': {'tcp_ports': [9926], 'allow_ptrace': True}, 'meta': {'service_name': 'ceph-exporter', 'ports': [9926], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0]\n', 'keyring': '[client.ceph-exporter.trial014]\nkey = AQAL0Y9pSQnMEBAAd1kOyDHxNKSk3p930enz8A==\n', 'prio-limit': '5', 'stats-period': '5'}} 2026-02-14 01:34:03,408 7efe3dbbce00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4' 2026-02-14 01:34:03,440 7efe3dbbce00 INFO Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-ceph-exporter-trial014 2026-02-14 01:34:03,440 7efe3dbbce00 INFO /usr/bin/podman: stderr Error: no such container "ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-ceph-exporter-trial014" 2026-02-14 01:34:03,456 7efe3dbbce00 INFO Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-ceph-exporter.trial014 2026-02-14 01:34:03,456 7efe3dbbce00 INFO /usr/bin/podman: stderr Error: no such container "ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-ceph-exporter.trial014" 2026-02-14 01:34:03,456 7efe3dbbce00 INFO Deploy daemon ceph-exporter.trial014 ... 2026-02-14 01:34:03,533 7efe3dbbce00 INFO Verifying port 0.0.0.0:9926 ... 2026-02-14 01:34:03,675 7efe3dbbce00 DEBUG Non-zero exit code 1 from systemctl reset-failed ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@ceph-exporter.trial014 2026-02-14 01:34:03,675 7efe3dbbce00 DEBUG systemctl: stderr Failed to reset failed state of unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@ceph-exporter.trial014.service: Unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@ceph-exporter.trial014.service not loaded. 2026-02-14 01:34:03,795 7efe3dbbce00 DEBUG systemctl: stderr Created symlink /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460.target.wants/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@ceph-exporter.trial014.service → /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@.service. 2026-02-14 01:34:03,866 7fe3214c2e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'orch', 'host', 'add', 'trial030'] 2026-02-14 01:34:03,947 7fe3214c2e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:03,947 7fe3214c2e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:03,947 7fe3214c2e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:03,948 7fe3214c2e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid orch host add trial030 2026-02-14 01:34:03,962 7efe3dbbce00 DEBUG firewalld.service is not enabled 2026-02-14 01:34:03,963 7efe3dbbce00 DEBUG Not possible to open ports <[9926]>. firewalld.service is not available 2026-02-14 01:34:04,094 7fe71f5d4e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:34:04,110 7fe71f5d4e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'crash.trial014', 'image': '', 'deploy_arguments': [], 'params': {'allow_ptrace': True}, 'meta': {'service_name': 'crash', 'ports': [], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0]\n', 'keyring': '[client.crash.trial014]\nkey = AQAL0Y9pPH3KOhAANHuSa4hyjyB/nHdHqJVEsg==\n'}} 2026-02-14 01:34:04,110 7fe71f5d4e00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4' 2026-02-14 01:34:04,141 7fe71f5d4e00 INFO Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-crash-trial014 2026-02-14 01:34:04,142 7fe71f5d4e00 INFO /usr/bin/podman: stderr Error: no such container "ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-crash-trial014" 2026-02-14 01:34:04,156 7fe71f5d4e00 INFO Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-crash.trial014 2026-02-14 01:34:04,157 7fe71f5d4e00 INFO /usr/bin/podman: stderr Error: no such container "ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-crash.trial014" 2026-02-14 01:34:04,157 7fe71f5d4e00 INFO Deploy daemon crash.trial014 ... 2026-02-14 01:34:04,372 7fe71f5d4e00 DEBUG Non-zero exit code 1 from systemctl reset-failed ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@crash.trial014 2026-02-14 01:34:04,372 7fe71f5d4e00 DEBUG systemctl: stderr Failed to reset failed state of unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@crash.trial014.service: Unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@crash.trial014.service not loaded. 2026-02-14 01:34:04,491 7fe71f5d4e00 DEBUG systemctl: stderr Created symlink /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460.target.wants/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@crash.trial014.service → /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@.service. 2026-02-14 01:34:04,755 7f904ec32e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.io/prometheus/node-exporter:v1.7.0', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:34:04,773 7f904ec32e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'node-exporter.trial014', 'image': '', 'deploy_arguments': [], 'params': {'tcp_ports': [9100], 'allow_ptrace': True}, 'meta': {'service_name': 'node-exporter', 'ports': [9100], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {}} 2026-02-14 01:34:04,773 7f904ec32e00 DEBUG Determined image: 'quay.io/prometheus/node-exporter:v1.7.0' 2026-02-14 01:34:04,805 7f904ec32e00 INFO Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-node-exporter-trial014 2026-02-14 01:34:04,805 7f904ec32e00 INFO /usr/bin/podman: stderr Error: no such container "ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-node-exporter-trial014" 2026-02-14 01:34:04,820 7f904ec32e00 INFO Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-node-exporter.trial014 2026-02-14 01:34:04,820 7f904ec32e00 INFO /usr/bin/podman: stderr Error: no such container "ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-node-exporter.trial014" 2026-02-14 01:34:04,820 7f904ec32e00 INFO Deploy daemon node-exporter.trial014 ... 2026-02-14 01:34:04,821 7f904ec32e00 INFO Verifying port 0.0.0.0:9100 ... 2026-02-14 01:34:04,968 7f904ec32e00 DEBUG Non-zero exit code 1 from systemctl reset-failed ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@node-exporter.trial014 2026-02-14 01:34:04,968 7f904ec32e00 DEBUG systemctl: stderr Failed to reset failed state of unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@node-exporter.trial014.service: Unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@node-exporter.trial014.service not loaded. 2026-02-14 01:34:05,092 7f904ec32e00 DEBUG systemctl: stderr Created symlink /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460.target.wants/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@node-exporter.trial014.service → /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@.service. 2026-02-14 01:34:05,787 7f904ec32e00 DEBUG firewalld.service is not enabled 2026-02-14 01:34:05,787 7f904ec32e00 DEBUG Not possible to open ports <[9100]>. firewalld.service is not available 2026-02-14 01:34:05,921 7fbb11071e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.io/prometheus/alertmanager:v0.25.0', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:34:05,940 7fbb11071e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'alertmanager.trial014', 'image': '', 'deploy_arguments': [], 'params': {'tcp_ports': [9093, 9094], 'allow_ptrace': True}, 'meta': {'service_name': 'alertmanager', 'ports': [9093, 9094], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'files': {'alertmanager.yml': "# This file is generated by cephadm.\n# See https://prometheus.io/docs/alerting/configuration/ for documentation.\n\nglobal:\n resolve_timeout: 5m\n http_config:\n tls_config:\n insecure_skip_verify: true\n\nroute:\n receiver: 'default'\n routes:\n - group_by: ['alertname']\n group_wait: 10s\n group_interval: 10s\n repeat_interval: 1h\n receiver: 'ceph-dashboard'\n\nreceivers:\n- name: 'default'\n webhook_configs:\n- name: 'ceph-dashboard'\n webhook_configs:\n"}, 'peers': []}} 2026-02-14 01:34:05,940 7fbb11071e00 DEBUG Determined image: 'quay.io/prometheus/alertmanager:v0.25.0' 2026-02-14 01:34:05,971 7fbb11071e00 INFO Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-alertmanager-trial014 2026-02-14 01:34:05,971 7fbb11071e00 INFO /usr/bin/podman: stderr Error: no such container "ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-alertmanager-trial014" 2026-02-14 01:34:05,986 7fe066e80e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'orch', 'host', 'ls', '--format=json'] 2026-02-14 01:34:05,986 7fbb11071e00 INFO Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-alertmanager.trial014 2026-02-14 01:34:05,986 7fbb11071e00 INFO /usr/bin/podman: stderr Error: no such container "ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-alertmanager.trial014" 2026-02-14 01:34:05,986 7fbb11071e00 INFO Deploy daemon alertmanager.trial014 ... 2026-02-14 01:34:06,041 7fe066e80e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:06,042 7fe066e80e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:06,042 7fe066e80e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:06,042 7fe066e80e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid orch host ls --format=json 2026-02-14 01:34:06,529 7f010d1fae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'osd', 'crush', 'tunables', 'default'] 2026-02-14 01:34:06,675 7f010d1fae00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:06,675 7f010d1fae00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:06,675 7f010d1fae00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:06,676 7f010d1fae00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid osd crush tunables default 2026-02-14 01:34:06,748 7fbb11071e00 INFO Verifying port 0.0.0.0:9093 ... 2026-02-14 01:34:06,749 7fbb11071e00 INFO Verifying port 0.0.0.0:9094 ... 2026-02-14 01:34:06,895 7fbb11071e00 DEBUG Non-zero exit code 1 from systemctl reset-failed ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@alertmanager.trial014 2026-02-14 01:34:06,895 7fbb11071e00 DEBUG systemctl: stderr Failed to reset failed state of unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@alertmanager.trial014.service: Unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@alertmanager.trial014.service not loaded. 2026-02-14 01:34:07,012 7fbb11071e00 DEBUG systemctl: stderr Created symlink /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460.target.wants/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@alertmanager.trial014.service → /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@.service. 2026-02-14 01:34:07,156 7fbb11071e00 DEBUG firewalld.service is not enabled 2026-02-14 01:34:07,156 7fbb11071e00 DEBUG Not possible to open ports <[9093, 9094]>. firewalld.service is not available 2026-02-14 01:34:07,355 7fd7f125fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.io/ceph/grafana:10.4.0', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:34:07,409 7fd7f125fe00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'grafana.trial014', 'image': '', 'deploy_arguments': [], 'params': {'tcp_ports': [3000], 'allow_ptrace': True}, 'meta': {'service_name': 'grafana', 'ports': [3000], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'files': {'grafana.ini': "# This file is generated by cephadm.\n[users]\n default_theme = light\n[auth.anonymous]\n enabled = true\n org_name = 'Main Org.'\n org_role = 'Viewer'\n[server]\n domain = 'bootstrap.storage.lab'\n protocol = https\n cert_file = /etc/grafana/certs/cert_file\n cert_key = /etc/grafana/certs/cert_key\n http_port = 3000\n http_addr = \n[snapshots]\n external_enabled = false\n[security]\n disable_initial_admin_creation = true\n cookie_secure = true\n cookie_samesite = none\n allow_embedding = true\n[analytics]\n check_for_updates = false\n reporting_enabled = false\n[plugins]\n check_for_plugin_updates = false\n public_key_retrieval_disabled = true", 'provisioning/datasources/ceph-dashboard.yml': "# This file is generated by cephadm.\napiVersion: 1\n\ndeleteDatasources:\n\ndatasources:\n\n - name: 'Loki'\n type: 'loki'\n access: 'proxy'\n url: ''\n basicAuth: false\n isDefault: false\n editable: false", 'certs/cert_file': '# generated by cephadm\n-----BEGIN CERTIFICATE-----\nMIICyDCCAbACEQDWFuGbwrFDSaO1W/2Qonk4MA0GCSqGSIb3DQEBDQUAMCIxDTAL\nBgNVBAoMBENlcGgxETAPBgNVBAMMCHRyaWFsMDE0MB4XDTI2MDIxNDAxMzQwN1oX\nDTM2MDIxMjAxMzQwN1owIjENMAsGA1UECgwEQ2VwaDERMA8GA1UEAwwIdHJpYWww\nMTQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDAZFJgUdruhEbDg2KF\nbh+5L7LQQ0IBiFrkI9MmBZ7pVetM7Yl+VyTuJeFQ6wYsQlUywcMPpLOEcafbDh9u\nnO/NRDKX/VhaCY9+deYMgQ+XoGA3a69jdllqJQKum/s5VleUOn8TmZ1rN/Rv1cCL\no0Ip0GKuG9xKJq3PoJlKDZzS/j0fJaTnQTZNGoszD9msA+7lZjZ9WBCAntAI0a0U\nxUCvAJT8YBG4KMSh2qXufWWOunvnmG8a9hDrq/IsVHowlbfgsQ+6tBMQIuo3OElg\n5j/JWlS4erC9tYdxrSSupJcR3LO9PrlUQ+cGDSe4sX3Yr+eACXQoyJUxXBj+CAUr\na0oXAgMBAAEwDQYJKoZIhvcNAQENBQADggEBAMAu3dnDu5qLUhUezg90qIXVP19g\nUeDirCGhpwZtoR2WpkLgh1Ceq8TK0nkHUkg3P/mwKTrI2Q5OD6UPyy40kyyx2t1k\nybbTzxo9Xz6F5jOpOuyoKz18M2U3fGDLxO5lbvlLCbeQez2o+AJhcFIv1M0CxGPB\nkbFj48cvz36cxtqSTmie+yl2UzKERQ++WzBhBUSWb1cntIAlbmCbVloseWGGcrmT\nBQ3qJYW3LgVCnSG5AwcajL0dLAhrhcvERBR2MCcprFQrGZhfxMBzMXoeEoIUXiOW\nZgSbAGEF9PrOAEWmAeZOLY1xqzA5P9Ww6p5I5UY8fTSorfqh4OVSGvNtCmM=\n-----END CERTIFICATE-----\n', 'certs/cert_key': '# generated by cephadm\n-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDAZFJgUdruhEbD\ng2KFbh+5L7LQQ0IBiFrkI9MmBZ7pVetM7Yl+VyTuJeFQ6wYsQlUywcMPpLOEcafb\nDh9unO/NRDKX/VhaCY9+deYMgQ+XoGA3a69jdllqJQKum/s5VleUOn8TmZ1rN/Rv\n1cCLo0Ip0GKuG9xKJq3PoJlKDZzS/j0fJaTnQTZNGoszD9msA+7lZjZ9WBCAntAI\n0a0UxUCvAJT8YBG4KMSh2qXufWWOunvnmG8a9hDrq/IsVHowlbfgsQ+6tBMQIuo3\nOElg5j/JWlS4erC9tYdxrSSupJcR3LO9PrlUQ+cGDSe4sX3Yr+eACXQoyJUxXBj+\nCAUra0oXAgMBAAECggEADotG+EXChtzJu1wCUFNGjf7/cI5fvgfrKcCqOf9GHR5D\n/yPHN3LVCt/NjGAdWQPcDE3o4L+IW/nPllgXTjW09yUxrJsYA8PMl43IH1KJCuMH\n2F4JUJE7fzJfD94g0XEn27/JqCypDC2bCp1JGgVJ1NQOXFp6KReKyjMPg4aglV6b\nG3UZp8LDBLzEzp3Qs5jhL9YCH+ETRr5tkrZFOFJsTSckMGmzCQ5QJKsz5tDrofo9\nyzMIDjkokdSUgc8UHuXDMjWkECWRpfcQoRNVeTQ+ZND0k4HXpJEk2ecqQKF4UK6f\nf75st1wEssyWYqGfUKZcKzfVzgfAHYyF2GV6h3FmSQKBgQD4pk5plfHs2PV1L0TB\nmCMeLOKngIZmy+foM/wT/gaJ/gXSaUJ1V2syZ1/igO2jZZppKMpEApb+vMPRMnRx\nsMymLHHIrwPoQkUgdtM6aGO1IoMFyPMs9gvQLYsZh5YsltulRhoxUKDobjkjujRq\nNCKKtqEP6cTlCUMYnQxqPdsdBQKBgQDGFEbEtAiB/c9ViL9dhUkIE3sahlv/dMtZ\n7tnFDFwx2PNjQ1KezUkGAktVYgxDve4D0uXE7TbYtRdOGZsbQd8KyBAsTstorfcT\nnjCZrp3Pi2cFrOcQzlKKeztNmJOhBV4XURlP26OckElaYNZMuejUxggbvXqGqnNA\nggTXUb3VawKBgQDauOgG5vhtgJfCYjcYGKzjkVY7Q8p2SG2SC5B1QaIc5o/We/MS\n5zgLmXV1+DWR4wFF+bSayLBsGj4iGkDl9UC/+B6LCrDkMM8/9GZwXX0/y8D3aMHI\nThMsJ7PRXDyUsos5wgWuVZ+FddfbfqM2vO+Zl3bsWZVyZ4hW8WcHWawRxQKBgEzA\nOpOpdsp3wEjSTeQ86wl4EvE6MpePd+QHCtaxbV/bQqOpyG/D95GeffY9oab2By3w\nIBvoFqJtuPQIM/Oh2Hzi/HFOrr6G+f/eAdLquDddOrQWRQpcB9uk2Fu+zSVNYSID\nyJ0K7CCUP/gHIcJlHvSjLy0ghEN99F6AgsOz5qy1AoGBAPa9+KxjN5k0Rbq7hSN6\noYd72KPE2hyC1rZfmOJBYn/a5a72tIahX2BNhgyXOmB65yZpYrDaxSsqxc6RFkQK\nFv9qwxpifEQ8/AwTNYkgbkLc2WXB9LT5QCCiuE7XcOyGdGfIxvdoGwke60Rkec6f\n+3ubSlyzhTdJGepyRdWYrXSG\n-----END PRIVATE KEY-----\n', 'provisioning/dashboards/default.yml': "# This file is generated by cephadm.\napiVersion: 1\n\nproviders:\n - name: 'Ceph Dashboard'\n orgId: 1\n folder: ''\n type: file\n disableDeletion: false\n updateIntervalSeconds: 3\n editable: false\n options:\n path: '/etc/grafana/provisioning/dashboards'", '/etc/grafana/provisioning/dashboards/host-details.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.3.2"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n },\n {\n "id": "singlestat",\n "name": "Singlestat",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "$ceph_hosts System Overview",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "format": "none",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 5,\n "w": 3,\n "x": 0,\n "y": 1\n },\n "id": 3,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "count(sum by (ceph_daemon) (ceph_osd_metadata{cluster=~\\"$cluster\\", hostname=~\\"$ceph_hosts\\"}))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "OSDs",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "% Utilization",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "percent"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 10,\n "w": 6,\n "x": 3,\n "y": 1\n },\n "id": 4,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "sum by (mode) (\\n rate(node_cpu{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\", mode=~\\"(irq|nice|softirq|steal|system|user|iowait)\\"}[$__rate_interval]) or\\n rate(node_cpu_seconds_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\", mode=~\\"(irq|nice|softirq|steal|system|user|iowait)\\"}[$__rate_interval])\\n) / (\\n scalar(\\n sum(rate(node_cpu{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\n rate(node_cpu_seconds_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]))\\n ) * 100\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{mode}}",\n "refId": "A"\n }\n ],\n "title": "CPU Utilization",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "RAM used",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "bytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 10,\n "w": 6,\n "x": 9,\n "y": 1\n },\n "id": 5,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "total",\n "color": "#bf1b00",\n "fill": 0,\n "linewidth": 2,\n "stack": false\n }\n ],\n "targets": [\n {\n "expr": "node_memory_MemFree{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\nnode_memory_MemFree_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Free",\n "refId": "A"\n },\n {\n "expr": "node_memory_MemTotal{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\nnode_memory_MemTotal_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "total",\n "refId": "B"\n },\n {\n "expr": "(\\n node_memory_Cached{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\n node_memory_Cached_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n) + (\\n node_memory_Buffers{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\n node_memory_Buffers_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n) + (\\n node_memory_Slab{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\n node_memory_Slab_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "buffers/cache",\n "refId": "C"\n },\n {\n "expr": "(\\n node_memory_MemTotal{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\n node_memory_MemTotal_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n) - (\\n (\\n node_memory_MemFree{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\n node_memory_MemFree_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n ) + (\\n node_memory_Cached{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\n node_memory_Cached_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n ) + (\\n node_memory_Buffers{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\n node_memory_Buffers_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n ) + (\\n node_memory_Slab{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\n node_memory_Slab_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n )\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "used",\n "refId": "D"\n }\n ],\n "title": "RAM Usage",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Send (-) / Receive (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "decbytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 10,\n "w": 6,\n "x": 15,\n "y": 1\n },\n "id": 6,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*tx/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "sum by (device) (\\n rate(node_network_receive_bytes{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\",device!=\\"lo\\"}[$__rate_interval]) or\\n rate(node_network_receive_bytes_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\",device!=\\"lo\\"}[$__rate_interval])\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}.rx",\n "refId": "A"\n },\n {\n "expr": "sum by (device) (\\n rate(node_network_transmit_bytes{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\",device!=\\"lo\\"}[$__rate_interval]) or\\n rate(node_network_transmit_bytes_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\",device!=\\"lo\\"}[$__rate_interval])\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}.tx",\n "refId": "B"\n }\n ],\n "title": "Network Load",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Send (-) / Receive (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "pps"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 5,\n "w": 3,\n "x": 21,\n "y": 1\n },\n "id": 7,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*tx/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "rate(node_network_receive_drop{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\nrate(node_network_receive_drop_total{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}.rx",\n "refId": "A"\n },\n {\n "expr": "rate(node_network_transmit_drop{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\nrate(node_network_transmit_drop_total{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}.tx",\n "refId": "B"\n }\n ],\n "title": "Network drop rate",\n "type": "timeseries"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Each OSD consists of a Journal/WAL partition and a data partition. The RAW Capacity shown is the sum of the data partitions across all OSDs on the selected OSD hosts.",\n "format": "bytes",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 5,\n "w": 3,\n "x": 0,\n "y": 6\n },\n "id": 8,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum(\\n ceph_osd_stat_bytes{cluster=~\\"$cluster\\", } and\\n on (ceph_daemon) ceph_disk_occupation{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\", cluster=~\\"$cluster\\", }\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Raw Capacity",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Send (-) / Receive (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "pps"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 5,\n "w": 3,\n "x": 21,\n "y": 6\n },\n "id": 9,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*tx/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "rate(node_network_receive_errs{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\nrate(node_network_receive_errs_total{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}.rx",\n "refId": "A"\n },\n {\n "expr": "rate(node_network_transmit_errs{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\nrate(node_network_transmit_errs_total{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}.tx",\n "refId": "B"\n }\n ],\n "title": "Network error rate",\n "type": "timeseries"\n },\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 11\n },\n "id": 10,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "OSD Disk Performance Statistics",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "ops"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 11,\n "x": 0,\n "y": 12\n },\n "id": 11,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*reads/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "label_replace(\\n (\\n rate(node_disk_writes_completed{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\n rate(node_disk_writes_completed_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) * on(instance, device) group_left(ceph_daemon) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{cluster=~\\"$cluster\\", }, \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}({{ceph_daemon}}) writes",\n "refId": "A"\n },\n {\n "expr": "label_replace(\\n (\\n rate(node_disk_reads_completed{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\n rate(node_disk_reads_completed_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) * on(instance, device) group_left(ceph_daemon) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{cluster=~\\"$cluster\\", }, \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}({{ceph_daemon}}) reads",\n "refId": "B"\n }\n ],\n "title": "$ceph_hosts Disk IOPS",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "Bps"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 11,\n "x": 12,\n "y": 12\n },\n "id": 12,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*read/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "label_replace(\\n (\\n rate(node_disk_bytes_written{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\n rate(node_disk_written_bytes_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) * on(instance, device)\\ngroup_left(ceph_daemon) label_replace(\\n label_replace(ceph_disk_occupation_human{cluster=~\\"$cluster\\", }, \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}({{ceph_daemon}}) write",\n "refId": "A"\n },\n {\n "expr": "label_replace(\\n (\\n rate(node_disk_bytes_read{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\n rate(node_disk_read_bytes_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) * on(instance, device)\\ngroup_left(ceph_daemon) label_replace(\\n label_replace(ceph_disk_occupation_human{cluster=~\\"$cluster\\", }, \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}({{ceph_daemon}}) read",\n "refId": "B"\n }\n ],\n "title": "$ceph_hosts Throughput by Disk",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "s"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 11,\n "x": 0,\n "y": 21\n },\n "id": 13,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "max by(instance, device) (\\n label_replace(\\n (\\n (rate(node_disk_write_time_seconds_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])) /\\n clamp_min(rate(node_disk_writes_completed_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]), 0.001) or\\n (rate(node_disk_read_time_seconds_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])) /\\n clamp_min(rate(node_disk_reads_completed_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]), 0.001)\\n ),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n )\\n) * on(instance, device) group_left(ceph_daemon) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"},\\n \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}({{ceph_daemon}})",\n "refId": "A"\n }\n ],\n "title": "$ceph_hosts Disk Latency",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "%Util",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "percent"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 11,\n "x": 12,\n "y": 21\n },\n "id": 14,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "label_replace(\\n (\\n (rate(node_disk_io_time_ms{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) / 10) or\\n rate(node_disk_io_time_seconds_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) * 100\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) * on(instance, device) group_left(ceph_daemon) label_replace(\\n label_replace(ceph_disk_occupation_human{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\", cluster=~\\"$cluster\\", },\\n \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}({{ceph_daemon}})",\n "refId": "A"\n }\n ],\n "title": "$ceph_hosts Disk utilization",\n "type": "timeseries"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "null",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "instance"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Instance"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Slow Ops"\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 0,\n "y": 30\n },\n "id": 15,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "10.4.0",\n "styles": "",\n "targets": [\n {\n "expr": "topk(10,\\n (sum by (instance)(ceph_daemon_health_metrics{type=\\"SLOW_OPS\\", ceph_daemon=~\\"osd.*\\", cluster=~\\"$cluster\\", }))\\n)\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Top Slow Ops per Host",\n "transformations": [\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time": true,\n "cluster": true\n },\n "includeByName": { },\n "indexByName": { },\n "renameByName": { }\n }\n }\n ],\n "type": "table"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin",\n "overview"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": null,\n "multi": false,\n "name": "ceph_hosts",\n "options": [ ],\n "query": "label_values({__name__=~\\"ceph_.+_metadata\\", cluster=~\\"$cluster\\", }, hostname)",\n "refresh": 1,\n "regex": "([^.]*).*",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "Host Details",\n "uid": "rtOg0AiWz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/radosgw-detail.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.0.0"\n },\n {\n "id": "grafana-piechart-panel",\n "name": "Pie Chart",\n "type": "panel",\n "version": "1.3.3"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "RGW Host Detail : $rgw_servers",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "s"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 0,\n "y": 1\n },\n "id": 3,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum by (instance_id) (\\n rate(ceph_rgw_op_get_obj_lat_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n rate(ceph_rgw_op_get_obj_lat_count{cluster=~\\"$cluster\\", }[$__rate_interval])\\n) * on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "GET {{ceph_daemon}}",\n "refId": "A"\n },\n {\n "expr": "sum by (instance_id) (\\n rate(ceph_rgw_op_put_obj_lat_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n rate(ceph_rgw_op_put_obj_lat_count{cluster=~\\"$cluster\\", }[$__rate_interval])\\n) * on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "PUT {{ceph_daemon}}",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "$rgw_servers GET/PUT Latencies",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "s",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "bytes"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 8,\n "w": 7,\n "x": 6,\n "y": 1\n },\n "id": 4,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "rate(ceph_rgw_op_get_obj_bytes{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "GETs {{ceph_daemon}}",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_rgw_op_put_obj_bytes{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon)\\n ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "PUTs {{ceph_daemon}}",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Bandwidth by HTTP Operation",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "bytes",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": {\n "GETs": "#7eb26d",\n "Other": "#447ebc",\n "PUTs": "#eab839",\n "Requests": "#3f2b5b",\n "Requests Failed": "#bf1b00"\n },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 8,\n "w": 7,\n "x": 13,\n "y": 1\n },\n "id": 5,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "rate(ceph_rgw_failed_req{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Requests Failed {{ceph_daemon}}",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_rgw_get{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "GETs {{ceph_daemon}}",\n "refId": "B"\n },\n {\n "expr": "rate(ceph_rgw_put{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "PUTs {{ceph_daemon}}",\n "refId": "C"\n },\n {\n "expr": "(\\n rate(ceph_rgw_req{cluster=~\\"$cluster\\", }[$__rate_interval]) -\\n (\\n rate(ceph_rgw_get{cluster=~\\"$cluster\\", }[$__rate_interval]) +\\n rate(ceph_rgw_put{cluster=~\\"$cluster\\", }[$__rate_interval])\\n )\\n) * on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Other {{ceph_daemon}}",\n "refId": "D"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "HTTP Request Breakdown",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n }\n },\n "mappings": [ ]\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "Failures"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "#bf1b00",\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "GETs"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "#7eb26d",\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Other (HEAD,POST,DELETE)"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "#447ebc",\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "PUTs"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "#eab839",\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Requests"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "#3f2b5b",\n "mode": "fixed"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 4,\n "x": 20,\n "y": 1\n },\n "id": 6,\n "options": {\n "displayLabels": [ ],\n "legend": {\n "calcs": [ ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "values": [\n "percent",\n "value"\n ]\n },\n "pieType": "pie",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "tooltip": {\n "mode": "single",\n "sort": "none"\n }\n },\n "targets": [\n {\n "expr": "rate(ceph_rgw_failed_req{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Failures {{ceph_daemon}}",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_rgw_get{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "GETs {{ceph_daemon}}",\n "refId": "B"\n },\n {\n "expr": "rate(ceph_rgw_put{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "PUTs {{ceph_daemon}}",\n "refId": "C"\n },\n {\n "expr": "(\\n rate(ceph_rgw_req{cluster=~\\"$cluster\\", }[$__rate_interval]) -\\n (\\n rate(ceph_rgw_get{cluster=~\\"$cluster\\", }[$__rate_interval]) +\\n rate(ceph_rgw_put{cluster=~\\"$cluster\\", }[$__rate_interval])\\n )\\n) * on (instance_id) group_left (ceph_daemon)\\n ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Other (DELETE,LIST) {{ceph_daemon}}",\n "refId": "D"\n }\n ],\n "title": "Workload Breakdown",\n "type": "piechart"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin",\n "overview"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": "",\n "multi": false,\n "name": "rgw_servers",\n "options": [ ],\n "query": "label_values(ceph_rgw_metadata{cluster=~\\"$cluster\\", }, ceph_daemon)",\n "refresh": 1,\n "regex": "",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "RGW Instance Detail",\n "uid": "x5ARzZtmk",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/radosgw-overview.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.0.0"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "RGW Overview - All Gateways",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "s"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 0,\n "y": 1\n },\n "id": 3,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "label_replace(\\n rate(ceph_rgw_op_get_obj_lat_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n rate(ceph_rgw_op_get_obj_lat_count{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", },\\n \\"rgw_host\\", \\"$1\\", \\"ceph_daemon\\", \\"rgw.(.*)\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "GET {{rgw_host}}",\n "refId": "A"\n },\n {\n "expr": "label_replace(\\n rate(ceph_rgw_op_put_obj_lat_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n rate(ceph_rgw_op_put_obj_lat_count{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", },\\n \\"rgw_host\\", \\"$1\\", \\"ceph_daemon\\", \\"rgw.(.*)\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "PUT {{rgw_host}}",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Average GET/PUT Latencies by RGW Instance",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "s",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "none"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 7,\n "x": 8,\n "y": 1\n },\n "id": 4,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum by (rgw_host) (\\n label_replace(\\n rate(ceph_rgw_req{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", },\\n \\"rgw_host\\", \\"$1\\", \\"ceph_daemon\\", \\"rgw.(.*)\\"\\n )\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{rgw_host}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Total Requests/sec by RGW Instance",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "none",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "Latencies are shown stacked, without a yaxis to provide a visual indication of GET latency imbalance across RGW hosts",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "s"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 6,\n "x": 15,\n "y": 1\n },\n "id": 5,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "label_replace(\\n rate(ceph_rgw_op_get_obj_lat_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n rate(ceph_rgw_op_get_obj_lat_count{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", },\\n \\"rgw_host\\", \\"$1\\", \\"ceph_daemon\\", \\"rgw.(.*)\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{rgw_host}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "GET Latencies by RGW Instance",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "s",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "Total bytes transferred in/out of all radosgw instances within the cluster",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "bytes"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 6,\n "w": 8,\n "x": 0,\n "y": 8\n },\n "id": 6,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(rate(ceph_rgw_op_get_obj_bytes{cluster=~\\"$cluster\\", }[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "GETs",\n "refId": "A"\n },\n {\n "expr": "sum(rate(ceph_rgw_op_put_obj_bytes{cluster=~\\"$cluster\\", }[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "PUTs",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Bandwidth Consumed by Type",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "bytes",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "Total bytes transferred in/out through get/put operations, by radosgw instance",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "bytes"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 6,\n "w": 7,\n "x": 8,\n "y": 8\n },\n "id": 7,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "label_replace(sum by (instance_id) (\\n rate(ceph_rgw_op_get_obj_bytes{cluster=~\\"$cluster\\", }[$__rate_interval]) +\\n rate(ceph_rgw_op_put_obj_bytes{cluster=~\\"$cluster\\", }[$__rate_interval])) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", },\\n \\"rgw_host\\", \\"$1\\", \\"ceph_daemon\\", \\"rgw.(.*)\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{rgw_host}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Bandwidth by RGW Instance",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "bytes",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "Latencies are shown stacked, without a yaxis to provide a visual indication of PUT latency imbalance across RGW hosts",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "s"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 6,\n "w": 6,\n "x": 15,\n "y": 8\n },\n "id": 8,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "label_replace(\\n rate(ceph_rgw_op_put_obj_lat_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n rate(ceph_rgw_op_put_obj_lat_count{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", },\\n \\"rgw_host\\", \\"$1\\", \\"ceph_daemon\\", \\"rgw.(.*)\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{rgw_host}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "PUT Latencies by RGW Instance",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "s",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 12,\n "w": 9,\n "x": 0,\n "y": 12\n },\n "id": 9,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "RGW Overview - HAProxy Metrics",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 12,\n "w": 5,\n "x": 0,\n "y": 12\n },\n "id": 10,\n "legend": {\n "alignAsTable": true,\n "avg": true,\n "current": true,\n "max": true,\n "min": true,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": true\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [\n [\n {\n "alias": "/.*Back.*/",\n "transform": "negative-Y"\n },\n {\n "alias": "/.*1.*/"\n },\n {\n "alias": "/.*2.*/"\n },\n {\n "alias": "/.*3.*/"\n },\n {\n "alias": "/.*4.*/"\n },\n {\n "alias": "/.*5.*/"\n },\n {\n "alias": "/.*other.*/"\n }\n ]\n ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(\\n rate(\\n haproxy_frontend_http_responses_total{code=~\\"$code\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\", proxy=~\\"frontend\\"}[$__rate_interval]\\n )\\n) by (code)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Frontend {{ code }}",\n "refId": "A"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_backend_http_responses_total{code=~\\"$code\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\", proxy=~\\"backend\\"}[$__rate_interval]\\n )\\n) by (code)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Backend {{ code }}",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Total responses by HTTP code",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 12,\n "w": 5,\n "x": 5,\n "y": 12\n },\n "id": 11,\n "legend": {\n "alignAsTable": true,\n "avg": true,\n "current": true,\n "max": true,\n "min": true,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": true\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [\n [\n {\n "alias": "/.*Response.*/",\n "transform": "negative-Y"\n },\n {\n "alias": "/.*Backend.*/",\n "transform": "negative-Y"\n }\n ]\n ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(\\n rate(\\n haproxy_frontend_http_requests_total{proxy=~\\"frontend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Requests",\n "refId": "A"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_backend_response_errors_total{proxy=~\\"backend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "Response errors",\n "refId": "B"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_frontend_request_errors_total{proxy=~\\"frontend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Requests errors",\n "refId": "C"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_backend_redispatch_warnings_total{proxy=~\\"backend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "Backend redispatch",\n "refId": "D"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_backend_retry_warnings_total{proxy=~\\"backend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "Backend retry",\n "refId": "E"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_frontend_requests_denied_total{proxy=~\\"frontend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "Request denied",\n "refId": "F"\n },\n {\n "expr": "sum(\\n haproxy_backend_current_queue{proxy=~\\"backend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "Backend Queued",\n "refId": "G"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Total requests / responses",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 12,\n "w": 5,\n "x": 10,\n "y": 12\n },\n "id": 12,\n "legend": {\n "alignAsTable": true,\n "avg": true,\n "current": true,\n "max": true,\n "min": true,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": true\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [\n [\n {\n "alias": "/.*Back.*/",\n "transform": "negative-Y"\n }\n ]\n ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(\\n rate(\\n haproxy_frontend_connections_total{proxy=~\\"frontend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Front",\n "refId": "A"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_backend_connection_attempts_total{proxy=~\\"backend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Back",\n "refId": "B"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_backend_connection_errors_total{proxy=~\\"backend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Back errors",\n "refId": "C"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Total number of connections",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 12,\n "w": 6,\n "x": 15,\n "y": 12\n },\n "id": 13,\n "legend": {\n "alignAsTable": true,\n "avg": true,\n "current": true,\n "max": true,\n "min": true,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": true\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [\n [\n {\n "alias": "/.*OUT.*/",\n "transform": "negative-Y"\n }\n ]\n ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(\\n rate(\\n haproxy_frontend_bytes_in_total{proxy=~\\"frontend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n ) * 8\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "IN Front",\n "refId": "A"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_frontend_bytes_out_total{proxy=~\\"frontend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n ) * 8\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "OUT Front",\n "refId": "B"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_backend_bytes_in_total{proxy=~\\"backend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n ) * 8\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "IN Back",\n "refId": "C"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_backend_bytes_out_total{proxy=~\\"backend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n ) * 8\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "OUT Back",\n "refId": "D"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Current total of incoming / outgoing bytes",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin",\n "overview"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": "",\n "multi": false,\n "name": "rgw_servers",\n "options": [ ],\n "query": "label_values(ceph_rgw_metadata{cluster=~\\"$cluster\\", }, ceph_daemon)",\n "refresh": 1,\n "regex": ".*",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": "HTTP Code",\n "multi": false,\n "name": "code",\n "options": [ ],\n "query": "label_values(haproxy_server_http_responses_total{job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}, code)",\n "refresh": 1,\n "regex": "",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": ".+",\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": "job haproxy",\n "multi": true,\n "name": "job_haproxy",\n "options": [ ],\n "query": "label_values(haproxy_server_status, job)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": "Ingress Service",\n "multi": false,\n "name": "ingress_service",\n "options": [ ],\n "query": "label_values(haproxy_server_status{job=~\\"$job_haproxy\\"}, instance)",\n "refresh": 1,\n "regex": "",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "RGW Overview",\n "uid": "WAkugZpiz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/rbd-details.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.3.3"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "Detailed Performance of RBD Images (IOPS/Throughput/Latency)",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "iops"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 9,\n "w": 8,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "rate(ceph_rbd_write_ops{pool=\\"$pool\\", image=\\"$image\\", cluster=~\\"$cluster\\", }[$__rate_interval])",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{pool}} Write",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_rbd_read_ops{pool=\\"$pool\\", image=\\"$image\\", cluster=~\\"$cluster\\", }[$__rate_interval])",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{pool}} Read",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "IOPS",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "iops",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "iops",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "Bps"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 9,\n "w": 8,\n "x": 8,\n "y": 0\n },\n "id": 3,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "rate(ceph_rbd_write_bytes{pool=\\"$pool\\", image=\\"$image\\", cluster=~\\"$cluster\\", }[$__rate_interval])",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{pool}} Write",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_rbd_read_bytes{pool=\\"$pool\\", image=\\"$image\\", cluster=~\\"$cluster\\", }[$__rate_interval])",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{pool}} Read",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Throughput",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "Bps",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "Bps",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "ns"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 9,\n "w": 8,\n "x": 16,\n "y": 0\n },\n "id": 4,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "rate(ceph_rbd_write_latency_sum{pool=\\"$pool\\", image=\\"$image\\", cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n rate(ceph_rbd_write_latency_count{pool=\\"$pool\\", image=\\"$image\\", cluster=~\\"$cluster\\", }[$__rate_interval])\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{pool}} Write",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_rbd_read_latency_sum{pool=\\"$pool\\", image=\\"$image\\", cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n rate(ceph_rbd_read_latency_count{pool=\\"$pool\\", image=\\"$image\\", cluster=~\\"$cluster\\", }[$__rate_interval])\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{pool}} Read",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Average Latency",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "ns",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "ns",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "",\n "multi": false,\n "name": "pool",\n "options": [ ],\n "query": "label_values(ceph_rbd_read_ops{cluster=~\\"$cluster\\", }, pool)",\n "refresh": 1,\n "regex": "",\n "sort": 0,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "",\n "multi": false,\n "name": "image",\n "options": [ ],\n "query": "label_values(ceph_rbd_read_ops{cluster=~\\"$cluster\\", pool=\\"$pool\\"}, image)",\n "refresh": 1,\n "regex": "",\n "sort": 0,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "RBD Details",\n "uid": "YhCYGcuZz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/rgw-s3-analytics.json': '{\n "__inputs": [ ],\n "__requires": [ ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "Overview",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "colors": null,\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "decbytes"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 0,\n "y": 1\n },\n "id": 3,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_put_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Total PUTs",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "decbytes"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 6,\n "y": 1\n },\n "id": 4,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum\\n(ceph_rgw_op_get_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Total GETs",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "none"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 12,\n "y": 1\n },\n "id": 5,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_put_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Total Objects",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "decbytes"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 18,\n "y": 1\n },\n "id": 6,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum\\n((sum by(instance_id)(ceph_rgw_op_put_obj_bytes) > 0) / (sum by(instance_id)(ceph_rgw_op_put_obj_ops) > 0) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Average Object Size",\n "transparent": false,\n "type": "stat"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 0,\n "y": 4\n },\n "id": 7,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_list_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "List Objects",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_list_buckets_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "List Buckets",\n "range": true,\n "refId": "B"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_put_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Put Objects",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_per_bucket_get_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Get Objects",\n "range": false,\n "refId": "D"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_del_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Delete Objects",\n "range": false,\n "refId": "E"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_del_bucket_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Delete Buckets",\n "range": false,\n "refId": "F"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_copy_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Copy Objects",\n "range": true,\n "refId": "G"\n }\n ],\n "title": "Total Operations",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "decbytes"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 8,\n "y": 4\n },\n "id": 8,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_put_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Put Objects",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_per_bucket_get_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Get Objects",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_del_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Delete Objects",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_copy_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Copy Objects",\n "range": true,\n "refId": "D"\n }\n ],\n "title": "Total Size",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "ms"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 16,\n "y": 4\n },\n "id": 9,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_list_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "List Object",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_list_buckets_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "List Bucket",\n "range": true,\n "refId": "B"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_put_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Put Object",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_get_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Get Object",\n "range": false,\n "refId": "D"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_del_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Delete Object",\n "range": false,\n "refId": "E"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_del_bucket_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Delete Bucket",\n "range": false,\n "refId": "F"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_copy_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Copy Object",\n "range": true,\n "refId": "G"\n }\n ],\n "title": "Total Latencies",\n "type": "bargauge"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "auto",\n "cellOptions": {\n "type": "color-text"\n },\n "filterable": false,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "options": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "decbytes"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 0,\n "y": 12\n },\n "id": 10,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true,\n "sortBy": [\n {\n "desc": true,\n "displayName": "PUTs"\n }\n ]\n },\n "pluginVersion": "9.4.7",\n "styles": "",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_put_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Upload Objects",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_get_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Get Objects",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_del_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Delete Objects",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_copy_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Copy Objects",\n "range": false,\n "refId": "D"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Summary Per Bucket by Bandwidth",\n "transformations": [\n {\n "id": "merge",\n "options": { }\n },\n {\n "id": "groupBy",\n "options": {\n "fields": {\n "Bucket": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #A": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #B": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #D": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #F": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "bucket": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "ceph_daemon": {\n "aggregations": [ ],\n "operation": "groupby"\n }\n }\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time 1": true,\n "Time 2": true,\n "Time 3": true,\n "Time 4": true,\n "Time 5": true,\n "Time 6": true,\n "Time 7": true,\n "__name__ 1": true,\n "__name__ 2": true,\n "__name__ 3": true,\n "__name__ 4": true,\n "__name__ 5": true,\n "__name__ 6": true,\n "__name__ 7": true,\n "ceph_daemon 1": false,\n "ceph_daemon 2": true,\n "ceph_daemon 3": true,\n "ceph_daemon 4": true,\n "instance 1": true,\n "instance 2": true,\n "instance 3": true,\n "instance 4": true,\n "instance 5": true,\n "instance 6": true,\n "instance 7": true,\n "instance_id 1": true,\n "instance_id 2": true,\n "instance_id 3": true,\n "instance_id 4": true,\n "instance_id 5": true,\n "instance_id 6": true,\n "instance_id 7": true,\n "job 1": true,\n "job 2": true,\n "job 3": true,\n "job 4": true,\n "job 5": true,\n "job 6": true,\n "job 7": true\n },\n "indexByName": {\n "Value #A": 2,\n "Value #B": 3,\n "Value #D": 4,\n "Value #F": 5,\n "bucket": 1,\n "ceph_daemon": 0\n },\n "renameByName": {\n "Bucket": "",\n "Value #A": "PUTs",\n "Value #B": "GETs",\n "Value #C": "List",\n "Value #D": "Delete",\n "Value #E": "Copy",\n "Value #F": "Copy",\n "Value #G": "",\n "bucket": "Bucket",\n "ceph_daemon": "Daemon",\n "ceph_daemon 1": "Daemon"\n }\n }\n }\n ],\n "type": "table"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "auto",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": false,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "options": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "ms"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 12,\n "y": 12\n },\n "id": 11,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true,\n "sortBy": [\n {\n "desc": true,\n "displayName": "PUTs"\n }\n ]\n },\n "pluginVersion": "9.4.7",\n "styles": "",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_bucket_list_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "List Objects",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_bucket_put_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Upload Objects",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_bucket_get_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Get Objects",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_bucket_del_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Delete Objects",\n "range": false,\n "refId": "D"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_bucket_copy_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Copy Objects",\n "range": false,\n "refId": "E"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Latency(ms) Per Bucket",\n "transformations": [\n {\n "id": "merge",\n "options": { }\n },\n {\n "id": "joinByField",\n "options": {\n "byField": "Bucket",\n "mode": "outer"\n }\n },\n {\n "id": "groupBy",\n "options": {\n "fields": {\n "Bucket": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #A": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #B": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #C": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #D": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #F": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "bucket": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "ceph_daemon": {\n "aggregations": [ ],\n "operation": "groupby"\n }\n }\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time 1": true,\n "Time 2": true,\n "Time 3": true,\n "Time 4": true,\n "Time 5": true,\n "Time 6": true,\n "Time 7": true,\n "__name__ 1": true,\n "__name__ 2": true,\n "__name__ 3": true,\n "__name__ 4": true,\n "__name__ 5": true,\n "__name__ 6": true,\n "__name__ 7": true,\n "ceph_daemon 1": true,\n "ceph_daemon 2": true,\n "ceph_daemon 3": true,\n "ceph_daemon 4": true,\n "ceph_daemon 5": true,\n "instance 1": true,\n "instance 2": true,\n "instance 3": true,\n "instance 4": true,\n "instance 5": true,\n "instance 6": true,\n "instance 7": true,\n "instance_id 1": true,\n "instance_id 2": true,\n "instance_id 3": true,\n "instance_id 4": true,\n "instance_id 5": true,\n "instance_id 6": true,\n "instance_id 7": true,\n "job 1": true,\n "job 2": true,\n "job 3": true,\n "job 4": true,\n "job 5": true,\n "job 6": true,\n "job 7": true\n },\n "indexByName": {\n "Value #A": 2,\n "Value #B": 3,\n "Value #C": 4,\n "Value #D": 5,\n "Value #F": 6,\n "bucket": 1,\n "ceph_daemon": 0\n },\n "renameByName": {\n "Bucket": "",\n "Value #A": "PUTs",\n "Value #B": "GETs",\n "Value #C": "List",\n "Value #D": "Delete",\n "Value #E": "Copy",\n "Value #F": "Copy",\n "Value #G": "",\n "bucket": "Bucket",\n "ceph_daemon": "Daemon"\n }\n }\n }\n ],\n "type": "table"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "auto",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": false,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "options": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "decbytes"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 0,\n "y": 20\n },\n "id": 12,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true,\n "sortBy": [\n {\n "desc": true,\n "displayName": "PUTs"\n }\n ]\n },\n "pluginVersion": "9.4.7",\n "styles": "",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_put_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Upload Objects",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_get_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Get Objects",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_del_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Delete Objects",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_copy_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Copy Objects",\n "range": false,\n "refId": "D"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Summary Per User By Bandwidth",\n "transformations": [\n {\n "id": "merge",\n "options": { }\n },\n {\n "id": "groupBy",\n "options": {\n "fields": {\n "User": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #A": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #B": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #D": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #F": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "ceph_daemon": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "instance": {\n "aggregations": [ ]\n },\n "user": {\n "aggregations": [ ],\n "operation": "groupby"\n }\n }\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time 1": true,\n "Time 2": true,\n "Time 3": true,\n "Time 4": true,\n "Time 5": true,\n "Time 6": true,\n "Time 7": true,\n "__name__ 1": true,\n "__name__ 2": true,\n "__name__ 3": true,\n "__name__ 4": true,\n "__name__ 5": true,\n "__name__ 6": true,\n "__name__ 7": true,\n "ceph_daemon 1": true,\n "ceph_daemon 2": true,\n "ceph_daemon 3": true,\n "ceph_daemon 4": true,\n "instance 1": true,\n "instance 2": true,\n "instance 3": true,\n "instance 4": true,\n "instance 5": true,\n "instance 6": true,\n "instance 7": true,\n "instance_id 1": true,\n "instance_id 2": true,\n "instance_id 3": true,\n "instance_id 4": true,\n "instance_id 5": true,\n "instance_id 6": true,\n "instance_id 7": true,\n "job 1": true,\n "job 2": true,\n "job 3": true,\n "job 4": true,\n "job 5": true,\n "job 6": true,\n "job 7": true\n },\n "indexByName": {\n "Value #A": 2,\n "Value #B": 3,\n "Value #D": 4,\n "Value #F": 5,\n "ceph_daemon": 0,\n "user": 1\n },\n "renameByName": {\n "Bucket": "",\n "Value #A": "PUTs",\n "Value #B": "GETs",\n "Value #C": "List",\n "Value #D": "Delete",\n "Value #E": "Copy",\n "Value #F": "Copy",\n "Value #G": "",\n "ceph_daemon": "Daemon",\n "user": "User"\n }\n }\n }\n ],\n "type": "table"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "auto",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": false,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "options": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "ms"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 12,\n "y": 20\n },\n "id": 13,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true,\n "sortBy": [\n {\n "desc": true,\n "displayName": "PUTs"\n }\n ]\n },\n "pluginVersion": "9.4.7",\n "styles": "",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_list_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_put_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_get_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_del_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "D"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_copy_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "E"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Latency(ms) Per User",\n "transformations": [\n {\n "id": "merge",\n "options": { }\n },\n {\n "id": "joinByField",\n "options": {\n "byField": "User",\n "mode": "outer"\n }\n },\n {\n "id": "groupBy",\n "options": {\n "fields": {\n "User": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #A": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #B": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #C": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #D": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #F": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "ceph_daemon": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "user": {\n "aggregations": [ ],\n "operation": "groupby"\n }\n }\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time 1": true,\n "Time 2": true,\n "Time 3": true,\n "Time 4": true,\n "Time 5": true,\n "Time 6": true,\n "Time 7": true,\n "__name__ 1": true,\n "__name__ 2": true,\n "__name__ 3": true,\n "__name__ 4": true,\n "__name__ 5": true,\n "__name__ 6": true,\n "__name__ 7": true,\n "ceph_daemon 1": true,\n "ceph_daemon 2": true,\n "ceph_daemon 3": true,\n "ceph_daemon 4": true,\n "ceph_daemon 5": true,\n "instance 1": true,\n "instance 2": true,\n "instance 3": true,\n "instance 4": true,\n "instance 5": true,\n "instance 6": true,\n "instance 7": true,\n "instance_id 1": true,\n "instance_id 2": true,\n "instance_id 3": true,\n "instance_id 4": true,\n "instance_id 5": true,\n "instance_id 6": true,\n "instance_id 7": true,\n "job 1": true,\n "job 2": true,\n "job 3": true,\n "job 4": true,\n "job 5": true,\n "job 6": true,\n "job 7": true\n },\n "indexByName": {\n "Value #A": 2,\n "Value #B": 3,\n "Value #C": 4,\n "Value #D": 5,\n "Value #F": 6,\n "ceph_daemon": 0,\n "user": 1\n },\n "renameByName": {\n "Bucket": "",\n "Value #A": "PUTs",\n "Value #B": "GETs",\n "Value #C": "List",\n "Value #D": "Delete",\n "Value #E": "Copy",\n "Value #F": "Copy",\n "Value #G": "",\n "ceph_daemon": "Daemon",\n "user": "User"\n }\n }\n }\n ],\n "type": "table"\n },\n {\n "collapse": false,\n "collapsed": true,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 28\n },\n "id": 14,\n "panels": [\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 0,\n "y": 29\n },\n "id": 15,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "topk(5, \\n sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_put_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })\\n)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Bucket PUTs by Operations",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 6,\n "y": 29\n },\n "id": 16,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "topk(5, \\n sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_get_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })\\n)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Bucket GETs by Operations",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 12,\n "y": 29\n },\n "id": 17,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [ ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "topk(5,\\n sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_put_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })\\n)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Buckets PUTs By Size",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 18,\n "y": 29\n },\n "id": 18,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [ ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "topk(5,\\n sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_get_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })\\n)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Buckets GETs By Size",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 0,\n "y": 37\n },\n "id": 19,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_put_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket PUTs by Size",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 6,\n "y": 37\n },\n "id": 20,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_get_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket GETs by Size",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 12,\n "y": 37\n },\n "id": 21,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_copy_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket Copy by Size",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 18,\n "y": 37\n },\n "id": 22,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_del_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket Delete by Size",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 0,\n "y": 45\n },\n "id": 23,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_get_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket GETs by Operations",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 6,\n "y": 45\n },\n "id": 24,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_put_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket PUTs by Operations",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 12,\n "y": 45\n },\n "id": 25,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_list_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket List by Operations",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 18,\n "y": 45\n },\n "id": 26,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_del_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket Delete by Operations",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 0,\n "y": 53\n },\n "id": 27,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_copy_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket Copy by Operations",\n "type": "timeseries"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "auto",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": false,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "options": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "none"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 12,\n "y": 53\n },\n "id": 28,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true,\n "sortBy": [\n {\n "desc": true,\n "displayName": "PUTs"\n }\n ]\n },\n "pluginVersion": "9.4.7",\n "styles": "",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_put_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_get_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_del_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_copy_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "D"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_list_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "E"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Summary Per Bucket by Operations",\n "transformations": [\n {\n "id": "merge",\n "options": { }\n },\n {\n "id": "joinByField",\n "options": {\n "byField": "Bucket",\n "mode": "outer"\n }\n },\n {\n "id": "groupBy",\n "options": {\n "fields": {\n "Bucket": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #A": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #B": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #C": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #D": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #F": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "bucket": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "ceph_daemon": {\n "aggregations": [ ],\n "operation": "groupby"\n }\n }\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time 1": true,\n "Time 2": true,\n "Time 3": true,\n "Time 4": true,\n "Time 5": true,\n "Time 6": true,\n "Time 7": true,\n "__name__": true,\n "__name__ 1": true,\n "__name__ 2": true,\n "__name__ 3": true,\n "__name__ 4": true,\n "__name__ 5": true,\n "__name__ 6": true,\n "__name__ 7": true,\n "ceph_daemon 1": true,\n "ceph_daemon 2": true,\n "ceph_daemon 3": true,\n "ceph_daemon 4": true,\n "instance 1": true,\n "instance 2": true,\n "instance 3": true,\n "instance 4": true,\n "instance 5": true,\n "instance 6": true,\n "instance 7": true,\n "instance_id 1": true,\n "instance_id 2": true,\n "instance_id 3": true,\n "instance_id 4": true,\n "instance_id 5": true,\n "instance_id 6": true,\n "instance_id 7": true,\n "job 1": true,\n "job 2": true,\n "job 3": true,\n "job 4": true,\n "job 5": true,\n "job 6": true,\n "job 7": true\n },\n "indexByName": {\n "Value #A": 2,\n "Value #B": 3,\n "Value #C": 4,\n "Value #D": 5,\n "Value #F": 6,\n "bucket": 1,\n "ceph_daemon": 0\n },\n "renameByName": {\n "Bucket": "",\n "Value #A": "PUTs",\n "Value #B": "GETs",\n "Value #C": "List",\n "Value #D": "Delete",\n "Value #E": "Copy",\n "Value #F": "Copy",\n "Value #G": "",\n "bucket": "Bucket",\n "ceph_daemon": "Daemon"\n }\n }\n }\n ],\n "type": "table"\n }\n ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "Buckets",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "collapse": false,\n "collapsed": true,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 29\n },\n "id": 29,\n "panels": [\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 0,\n "y": 62\n },\n "id": 30,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "topk(5, \\n sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_put_obj_ops ) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })\\n)\\n",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Users PUTs By Operations",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 6,\n "y": 62\n },\n "id": 31,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "topk(5, \\n sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_get_obj_ops ) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })\\n)\\n",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Users GETs by Operations",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 12,\n "y": 62\n },\n "id": 32,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [ ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "topk(5, \\n sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_put_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })\\n)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Users PUTs by Size",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 18,\n "y": 62\n },\n "id": 33,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [ ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "topk(5, \\n sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_get_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })\\n)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Users GETs By Size",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 0,\n "y": 70\n },\n "id": 34,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_put_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User PUTs by Size",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 6,\n "y": 70\n },\n "id": 35,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_get_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User GETs by Size",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 12,\n "y": 70\n },\n "id": 36,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_del_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User Delete by Size",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 18,\n "y": 70\n },\n "id": 37,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_copy_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User COPY by Size",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 0,\n "y": 78\n },\n "id": 38,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_get_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User GETs by Operations",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 6,\n "y": 78\n },\n "id": 39,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_put_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User PUTs by Operations",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 12,\n "y": 78\n },\n "id": 40,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_list_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User List by Operations",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 18,\n "y": 78\n },\n "id": 41,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_del_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User Delete by Operations",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 0,\n "y": 86\n },\n "id": 42,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_copy_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User Copy by Operations",\n "type": "timeseries"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "auto",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": false,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "options": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "none"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 12,\n "y": 86\n },\n "id": 43,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true,\n "sortBy": [\n {\n "desc": true,\n "displayName": "PUTs"\n }\n ]\n },\n "pluginVersion": "9.4.7",\n "styles": "",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (user, ceph_daemon) (ceph_rgw_op_per_user_put_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (user, ceph_daemon) (ceph_rgw_op_per_user_get_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (user, ceph_daemon) (ceph_rgw_op_per_user_del_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (user, ceph_daemon) (ceph_rgw_op_per_user_copy_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "D"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (user, ceph_daemon) (ceph_rgw_op_per_user_list_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "E"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Summary Per User By Operations",\n "transformations": [\n {\n "id": "merge",\n "options": { }\n },\n {\n "id": "joinByField",\n "options": {\n "byField": "User",\n "mode": "outer"\n }\n },\n {\n "id": "groupBy",\n "options": {\n "fields": {\n "User": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #A": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #B": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #C": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #D": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #F": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "ceph_daemon": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "user": {\n "aggregations": [ ],\n "operation": "groupby"\n }\n }\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": { },\n "indexByName": {\n "Value #A": 2,\n "Value #B": 3,\n "Value #C": 4,\n "Value #D": 5,\n "Value #F": 6,\n "ceph_daemon": 0,\n "user": 1\n },\n "renameByName": {\n "Value #A": "PUTs",\n "Value #B": "GETs",\n "Value #C": "LIST",\n "Value #D": "DELETE",\n "Value #F": "COPY",\n "ceph_daemon": "Daemon",\n "user": "User"\n }\n }\n }\n ],\n "type": "table"\n }\n ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "Users",\n "titleSize": "h6",\n "type": "row"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 22,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": null,\n "multi": false,\n "name": "rgw_servers",\n "options": [ ],\n "query": "label_values(ceph_rgw_metadata{cluster=~\\"$cluster\\", }, ceph_daemon)",\n "refresh": 2,\n "regex": "",\n "sort": 0,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "datasource": "$datasource",\n "hide": 2,\n "label": "filters",\n "name": "Filters",\n "type": "adhoc"\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "RGW S3 Analytics",\n "uid": "BnxelG7Sz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/hosts-overview.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.3.2"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n },\n {\n "id": "singlestat",\n "name": "Singlestat",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "format": "none",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 5,\n "w": 4,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "count(sum by (hostname) (ceph_osd_metadata{cluster=~\\"$cluster\\", }))",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "OSD Hosts",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Average CPU busy across all hosts (OSD, RGW, MON etc) within the cluster",\n "format": "percentunit",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 5,\n "w": 4,\n "x": 4,\n "y": 0\n },\n "id": 3,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "avg(1 - (\\n avg by(instance) (\\n rate(node_cpu_seconds_total{mode=\'idle\',instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}[$__rate_interval]) or\\n rate(node_cpu{mode=\'idle\',instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}[$__rate_interval])\\n )\\n))\\n",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "AVG CPU Busy",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Average Memory Usage across all hosts in the cluster (excludes buffer/cache usage)",\n "format": "percentunit",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 5,\n "w": 4,\n "x": 8,\n "y": 0\n },\n "id": 4,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "avg ((\\n (\\n node_memory_MemTotal{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"} or\\n node_memory_MemTotal_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}\\n ) - ((\\n node_memory_MemFree{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"} or\\n node_memory_MemFree_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}) +\\n (\\n node_memory_Cached{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"} or\\n node_memory_Cached_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}\\n ) + (\\n node_memory_Buffers{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"} or\\n node_memory_Buffers_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}\\n ) + (\\n node_memory_Slab{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"} or\\n node_memory_Slab_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}\\n )\\n )\\n) / (\\n node_memory_MemTotal{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"} or\\n node_memory_MemTotal_bytes{instance=~\\"($osd_hosts|$rgw_hosts|$mon_hosts|$mds_hosts).*\\"}\\n))\\n",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "AVG RAM Utilization",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "IOPS Load at the device as reported by the OS on all OSD hosts",\n "format": "none",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 5,\n "w": 4,\n "x": 12,\n "y": 0\n },\n "id": 5,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum ((\\n rate(node_disk_reads_completed{instance=~\\"($osd_hosts).*\\"}[$__rate_interval]) or\\n rate(node_disk_reads_completed_total{instance=~\\"($osd_hosts).*\\"}[$__rate_interval])\\n) + (\\n rate(node_disk_writes_completed{instance=~\\"($osd_hosts).*\\"}[$__rate_interval]) or\\n rate(node_disk_writes_completed_total{instance=~\\"($osd_hosts).*\\"}[$__rate_interval])\\n))\\n",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Physical IOPS",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Average Disk utilization for all OSD data devices (i.e. excludes journal/WAL)",\n "format": "percent",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 5,\n "w": 4,\n "x": 16,\n "y": 0\n },\n "id": 6,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "avg (\\n label_replace(\\n (rate(node_disk_io_time_ms[$__rate_interval]) / 10 ) or\\n (rate(node_disk_io_time_seconds_total[$__rate_interval]) * 100),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^.:]*).*\\"\\n ) * on(instance, device) group_left(ceph_daemon) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{instance=~\\"($osd_hosts).*\\", cluster=~\\"$cluster\\", },\\n \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^.:]*).*\\"\\n )\\n)\\n",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "AVG Disk Utilization",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Total send/receive network load across all hosts in the ceph cluster",\n "format": "bytes",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 5,\n "w": 4,\n "x": 20,\n "y": 0\n },\n "id": 7,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum (\\n (\\n rate(node_network_receive_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\",device!=\\"lo\\"}[$__rate_interval]) or\\n rate(node_network_receive_bytes_total{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\",device!=\\"lo\\"}[$__rate_interval])\\n ) unless on (device, instance)\\n label_replace((node_bonding_slaves > 0), \\"device\\", \\"$1\\", \\"master\\", \\"(.+)\\")\\n) +\\nsum (\\n (\\n rate(node_network_transmit_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\",device!=\\"lo\\"}[$__rate_interval]) or\\n rate(node_network_transmit_bytes_total{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\",device!=\\"lo\\"}[$__rate_interval])\\n ) unless on (device, instance)\\n label_replace((node_bonding_slaves > 0), \\"device\\", \\"$1\\", \\"master\\", \\"(.+)\\")\\n)\\n",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Network Load",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "percent"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 0,\n "y": 5\n },\n "id": 8,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "topk(10,\\n 100 * (\\n 1 - (\\n avg by(instance) (\\n rate(node_cpu_seconds_total{mode=\'idle\',instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}[$__rate_interval]) or\\n rate(node_cpu{mode=\'idle\',instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}[$__rate_interval])\\n )\\n )\\n )\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{instance}}",\n "refId": "A"\n }\n ],\n "title": "CPU Busy - Top 10 Hosts",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "Bps"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 12,\n "y": 5\n },\n "id": 9,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "topk(10, (sum by(instance) (\\n (\\n rate(node_network_receive_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\",device!=\\"lo\\"}[$__rate_interval]) or\\n rate(node_network_receive_bytes_total{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\",device!=\\"lo\\"}[$__rate_interval])\\n ) +\\n (\\n rate(node_network_transmit_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\",device!=\\"lo\\"}[$__rate_interval]) or\\n rate(node_network_transmit_bytes_total{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\",device!=\\"lo\\"}[$__rate_interval])\\n ) unless on (device, instance)\\n label_replace((node_bonding_slaves > 0), \\"device\\", \\"$1\\", \\"master\\", \\"(.+)\\"))\\n))\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{instance}}",\n "refId": "A"\n }\n ],\n "title": "Network Load - Top 10 Hosts",\n "type": "timeseries"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": null,\n "multi": false,\n "name": "osd_hosts",\n "options": [ ],\n "query": "label_values(ceph_osd_metadata{cluster=~\\"$cluster\\", }, hostname)",\n "refresh": 1,\n "regex": "([^.]*).*",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": null,\n "multi": false,\n "name": "mon_hosts",\n "options": [ ],\n "query": "label_values(ceph_mon_metadata{cluster=~\\"$cluster\\", }, hostname)",\n "refresh": 1,\n "regex": "mon.(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": null,\n "multi": false,\n "name": "mds_hosts",\n "options": [ ],\n "query": "label_values(ceph_mds_inodes{cluster=~\\"$cluster\\"}, hostname)",\n "refresh": 1,\n "regex": "mds.(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": null,\n "multi": false,\n "name": "rgw_hosts",\n "options": [ ],\n "query": "label_values(ceph_rgw_metadata{cluster=~\\"$cluster\\"}, hostname)",\n "refresh": 1,\n "regex": "rgw.(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "Host Overview",\n "uid": "y0KGL0iZz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/pool-overview.json': '{\n "__inputs": [ ],\n "__requires": [ ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "format": "none",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "count(ceph_pool_metadata{cluster=~\\"$cluster\\", })",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Pools",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "avg"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Count of the pools that have compression enabled",\n "format": "none",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 3,\n "y": 0\n },\n "id": 3,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "count(ceph_pool_metadata{compression_mode!=\\"none\\", cluster=~\\"$cluster\\", })",\n "format": "",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Pools with Compression",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Total raw capacity available to the cluster",\n "format": "bytes",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 6,\n "y": 0\n },\n "id": 4,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum(ceph_osd_stat_bytes{cluster=~\\"$cluster\\", })",\n "format": "",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Total Raw Capacity",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Total raw capacity consumed by user data and associated overheads (metadata + redundancy)",\n "format": "bytes",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 9,\n "y": 0\n },\n "id": 5,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum(ceph_pool_bytes_used{cluster=~\\"$cluster\\", })",\n "format": "",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Raw Capacity Consumed",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Total of client data stored in the cluster",\n "format": "bytes",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 12,\n "y": 0\n },\n "id": 6,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum(ceph_pool_stored{cluster=~\\"$cluster\\", })",\n "format": "",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Logical Stored ",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "A compression saving is determined as the data eligible to be compressed minus the capacity used to store the data after compression",\n "format": "bytes",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 15,\n "y": 0\n },\n "id": 7,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum(\\n ceph_pool_compress_under_bytes{cluster=~\\"$cluster\\", } -\\n ceph_pool_compress_bytes_used{cluster=~\\"$cluster\\", }\\n)\\n",\n "format": "",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Compression Savings",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Indicates how suitable the data is within the pools that are/have been enabled for compression - averaged across all pools holding compressed data",\n "format": "percent",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 18,\n "y": 0\n },\n "id": 8,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "(\\n sum(ceph_pool_compress_under_bytes{cluster=~\\"$cluster\\", } > 0) /\\n sum(ceph_pool_stored_raw{cluster=~\\"$cluster\\", } and ceph_pool_compress_under_bytes{cluster=~\\"$cluster\\", } > 0)\\n) * 100\\n",\n "format": "table",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Compression Eligibility",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "This factor describes the average ratio of data eligible to be compressed divided by the data actually stored. It does not account for data written that was ineligible for compression (too small, or compression yield too low)",\n "format": "none",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 21,\n "y": 0\n },\n "id": 9,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum(\\n ceph_pool_compress_under_bytes{cluster=~\\"$cluster\\", } > 0)\\n / sum(ceph_pool_compress_bytes_used{cluster=~\\"$cluster\\", } > 0\\n)\\n",\n "format": "",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Compression Factor",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "auto",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "Time"\n },\n "properties": [\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "instance"\n },\n "properties": [\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "job"\n },\n "properties": [\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "name"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Pool Name"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "pool_id"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Pool ID"\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #A"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Compression Factor"\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #D"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "% Used"\n },\n {\n "id": "unit",\n "value": "percentunit"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.cellOptions",\n "value": {\n "type": "color-text"\n }\n },\n {\n "id": "thresholds",\n "value": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(245, 54, 54, 0.9)",\n "value": null\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 70\n },\n {\n "color": "rgba(50, 172, 45, 0.97)",\n "value": 85\n }\n ]\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #B"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Usable Free"\n },\n {\n "id": "unit",\n "value": "bytes"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #C"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Compression Eligibility"\n },\n {\n "id": "unit",\n "value": "percent"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #E"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Compression Savings"\n },\n {\n "id": "unit",\n "value": "bytes"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #F"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Growth (5d)"\n },\n {\n "id": "unit",\n "value": "bytes"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.cellOptions",\n "value": {\n "type": "color-text"\n }\n },\n {\n "id": "thresholds",\n "value": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(245, 54, 54, 0.9)",\n "value": null\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 70\n },\n {\n "color": "rgba(50, 172, 45, 0.97)",\n "value": 85\n }\n ]\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #G"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "IOPS"\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #H"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Bandwidth"\n },\n {\n "id": "unit",\n "value": "Bps"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "__name__"\n },\n "properties": [\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "type"\n },\n "properties": [\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "compression_mode"\n },\n "properties": [\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "description"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Type"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #J"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Stored"\n },\n {\n "id": "unit",\n "value": "bytes"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #I"\n },\n "properties": [\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #K"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Compression"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 6,\n "w": 24,\n "x": 0,\n "y": 3\n },\n "id": 10,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "10.4.0",\n "styles": "",\n "targets": [\n {\n "expr": "(\\n ceph_pool_compress_under_bytes{cluster=~\\"$cluster\\", } /\\n ceph_pool_compress_bytes_used{cluster=~\\"$cluster\\", } > 0\\n) and on(pool_id) (\\n (\\n (ceph_pool_compress_under_bytes{cluster=~\\"$cluster\\", } > 0) /\\n ceph_pool_stored_raw{cluster=~\\"$cluster\\", }\\n ) * 100 > 0.5\\n)\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "A",\n "refId": "A"\n },\n {\n "expr": "ceph_pool_max_avail{cluster=~\\"$cluster\\", } *\\n on(pool_id) group_left(name) ceph_pool_metadata{cluster=~\\"$cluster\\", }\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "B",\n "refId": "B"\n },\n {\n "expr": "(\\n (ceph_pool_compress_under_bytes{cluster=~\\"$cluster\\", } > 0) /\\n ceph_pool_stored_raw{cluster=~\\"$cluster\\", }\\n) * 100\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "C",\n "refId": "C"\n },\n {\n "expr": "ceph_pool_percent_used{cluster=~\\"$cluster\\", } *\\n on(pool_id) group_left(name) ceph_pool_metadata{cluster=~\\"$cluster\\", }\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "D",\n "refId": "D"\n },\n {\n "expr": "ceph_pool_compress_under_bytes{cluster=~\\"$cluster\\", } -\\n ceph_pool_compress_bytes_used{cluster=~\\"$cluster\\", } > 0\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "E",\n "refId": "E"\n },\n {\n "expr": "delta(ceph_pool_stored{cluster=~\\"$cluster\\", }[5d])",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "F",\n "refId": "F"\n },\n {\n "expr": "rate(ceph_pool_rd{cluster=~\\"$cluster\\", }[$__rate_interval])\\n + rate(ceph_pool_wr{cluster=~\\"$cluster\\", }[$__rate_interval])\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "G",\n "refId": "G"\n },\n {\n "expr": "rate(ceph_pool_rd_bytes{cluster=~\\"$cluster\\", }[$__rate_interval]) +\\n rate(ceph_pool_wr_bytes{cluster=~\\"$cluster\\", }[$__rate_interval])\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "H",\n "refId": "H"\n },\n {\n "expr": "ceph_pool_metadata{cluster=~\\"$cluster\\", }",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "I",\n "refId": "I"\n },\n {\n "expr": "ceph_pool_stored{cluster=~\\"$cluster\\", } * on(pool_id) group_left ceph_pool_metadata{cluster=~\\"$cluster\\", }",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "J",\n "refId": "J"\n },\n {\n "expr": "ceph_pool_metadata{compression_mode!=\\"none\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "K",\n "refId": "K"\n },\n {\n "expr": "",\n "format": "",\n "intervalFactor": "",\n "legendFormat": "L",\n "refId": "L"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Pool Overview",\n "transformations": [\n {\n "id": "merge",\n "options": { }\n },\n {\n "id": "seriesToRows",\n "options": { }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time": true,\n "Value #A": true,\n "Value #B": false,\n "Value #C": true,\n "Value #D": false,\n "Value #E": true,\n "Value #I": true,\n "Value #K": true,\n "__name__": true,\n "cluster": true,\n "compression_mode": true,\n "instance": true,\n "job": true,\n "pool_id": true,\n "type": true\n },\n "includeByName": { },\n "indexByName": { },\n "renameByName": { }\n }\n }\n ],\n "type": "table"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "IOPS",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 0,\n "y": 9\n },\n "id": 11,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "topk($topk,\\n round(\\n (\\n rate(ceph_pool_rd{cluster=~\\"$cluster\\", }[$__rate_interval]) +\\n rate(ceph_pool_wr{cluster=~\\"$cluster\\", }[$__rate_interval])\\n ), 1\\n ) * on(pool_id) group_left(instance,name) ceph_pool_metadata{cluster=~\\"$cluster\\", }\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "refId": "A"\n },\n {\n "expr": "topk($topk,\\n rate(ceph_pool_wr{cluster=~\\"$cluster\\", }[$__rate_interval]) +\\n on(pool_id) group_left(instance,name) ceph_pool_metadata{cluster=~\\"$cluster\\", }\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{name}} - write",\n "refId": "B"\n }\n ],\n "title": "Top $topk Client IOPS by Pool",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Throughput",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "Bps"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 12,\n "y": 9\n },\n "id": 12,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "topk($topk,\\n (\\n rate(ceph_pool_rd_bytes{cluster=~\\"$cluster\\", }[$__rate_interval]) +\\n rate(ceph_pool_wr_bytes{cluster=~\\"$cluster\\", }[$__rate_interval])\\n ) * on(pool_id) group_left(instance, name) ceph_pool_metadata{cluster=~\\"$cluster\\", }\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "refId": "A"\n }\n ],\n "title": "Top $topk Client Bandwidth by Pool",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Capacity Used",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "bytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 24,\n "x": 0,\n "y": 17\n },\n "id": 13,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "ceph_pool_bytes_used{cluster=~\\"$cluster\\", } * on(pool_id) group_right ceph_pool_metadata{cluster=~\\"$cluster\\", }",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "refId": "A"\n }\n ],\n "title": "Pool Capacity Usage (RAW)",\n "type": "timeseries"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 22,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": {\n "text": "15",\n "value": "15"\n },\n "hide": 0,\n "includeAll": false,\n "label": "TopK",\n "multi": false,\n "name": "topk",\n "options": [\n {\n "text": "15",\n "value": "15"\n }\n ],\n "query": "15",\n "refresh": 0,\n "type": "custom"\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "Ceph Pools Overview",\n "uid": "z99hzWtmk",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/radosgw-sync-overview.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.0.0"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "Bps"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": true,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum by (source_zone) (rate(ceph_data_sync_from_zone_fetch_bytes_sum{cluster=~\\"$cluster\\", }[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{source_zone}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Replication (throughput) from Source Zone",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "Bps",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 8,\n "y": 0\n },\n "id": 3,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": true,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum by (source_zone) (rate(ceph_data_sync_from_zone_fetch_bytes_count{cluster=~\\"$cluster\\", }[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{source_zone}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Replication (objects) from Source Zone",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": "Objects/s",\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "ms"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 16,\n "y": 0\n },\n "id": 4,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": true,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum by (source_zone) (rate(ceph_data_sync_from_zone_poll_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{source_zone}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Polling Request Latency from Source Zone",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "ms",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 0,\n "y": 7\n },\n "id": 5,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": true,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum by (source_zone) (rate(ceph_data_sync_from_zone_fetch_errors{cluster=~\\"$cluster\\", }[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{source_zone}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Unsuccessful Object Replications from Source Zone",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": "Count/s",\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "s"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 16,\n "x": 8,\n "y": 7\n },\n "id": 6,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "right",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "rate(ceph_rgw_sync_delta_sync_delta[$__rate_interval])",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{instance_id}} - {{shard_id}}",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Replication(Time) Delta per shard",\n "type": "timeseries"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin",\n "overview"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": null,\n "multi": false,\n "name": "rgw_servers",\n "options": [ ],\n "query": "label_values(ceph_rgw_metadata{cluster=~\\"$cluster\\", }, ceph_daemon)",\n "refresh": 1,\n "regex": "rgw.(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "RGW Sync Overview",\n "uid": "rgw-sync-overview",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/ceph-cluster-advanced.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.3.2"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n },\n {\n "id": "heatmap",\n "name": "Heatmap",\n "type": "panel",\n "version": "5.0.0"\n },\n {\n "id": "singlestat",\n "name": "Singlestat",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "Ceph cluster overview",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "CLUSTER STATE",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "0": {\n "text": "HEALTHY"\n },\n "1": {\n "text": "WARNING"\n },\n "2": {\n "text": "ERROR"\n }\n },\n "type": "value"\n },\n {\n "id": 1,\n "options": {\n "match": null,\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "#9ac48a"\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 1\n },\n {\n "color": "rgba(245, 54, 54, 0.9)",\n "value": 2\n }\n ]\n },\n "unit": "none"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 0,\n "y": 1\n },\n "id": 3,\n "interval": "1m",\n "links": [ ],\n "options": {\n "colorMode": "value",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "ceph_health_status{cluster=~\\"$cluster\\", }",\n "format": "time_series",\n "instant": true,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Ceph health status",\n "transparent": true,\n "type": "stat"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": null,\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "max": 1,\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(245, 54, 54, 0.9)"\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 0.10000000000000001\n },\n {\n "color": "rgba(50, 172, 45, 0.97)",\n "value": 0.29999999999999999\n }\n ]\n },\n "unit": "percentunit"\n }\n },\n "gridPos": {\n "h": 6,\n "w": 3,\n "x": 3,\n "y": 1\n },\n "id": 4,\n "interval": "1m",\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "showThresholdLabels": false,\n "showThresholdMarkers": true\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "(ceph_cluster_total_bytes{cluster=~\\"$cluster\\", }-ceph_cluster_total_used_bytes{cluster=~\\"$cluster\\", })/ceph_cluster_total_bytes{cluster=~\\"$cluster\\", }",\n "format": "time_series",\n "instant": true,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Available Capacity",\n "transparent": false,\n "type": "gauge"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 2,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": null,\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(50, 172, 45, 0.97)"\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 0.025000000000000001\n },\n {\n "color": "rgba(245, 54, 54, 0.9)",\n "value": 1\n }\n ]\n },\n "unit": "bytes"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 6,\n "y": 1\n },\n "id": 5,\n "interval": "1m",\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "ceph_cluster_total_bytes{cluster=~\\"$cluster\\", }",\n "format": "time_series",\n "instant": true,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Cluster Capacity",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 1,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": null,\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "Bps"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 9,\n "y": 1\n },\n "id": 6,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(irate(ceph_osd_op_w_in_bytes{cluster=~\\"$cluster\\", }[5m]))",\n "format": "time_series",\n "instant": true,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "Write Throughput",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 1,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": null,\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "#d44a3a"\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 0\n },\n {\n "color": "#9ac48a",\n "value": 0\n }\n ]\n },\n "unit": "Bps"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 12,\n "y": 1\n },\n "id": 7,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(irate(ceph_osd_op_r_out_bytes{cluster=~\\"$cluster\\", }[5m]))",\n "format": "time_series",\n "instant": true,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "Read Throughput",\n "transparent": false,\n "type": "stat"\n },\n {\n "colorMode": "Panel",\n "colors": {\n "crit": "rgb(255, 0, 0)",\n "disable": "rgba(128, 128, 128, 0.9)",\n "ok": "rgba(50, 128, 45, 0.9)",\n "warn": "rgba(237, 129, 40, 0.9)"\n },\n "cornerRadius": 0,\n "datasource": "$datasource",\n "description": "",\n "displayName": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "All"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Out"\n },\n "properties": [\n {\n "id": "thresholds",\n "value": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 1\n }\n ]\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Down"\n },\n "properties": [\n {\n "id": "thresholds",\n "value": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 1\n }\n ]\n }\n }\n ]\n }\n ]\n },\n "flipCard": false,\n "flipTime": 5,\n "fontFormat": "Regular",\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 15,\n "y": 1\n },\n "id": 8,\n "isAutoScrollOnOverflow": false,\n "isGrayOnNoData": false,\n "isHideAlertsOnDisable": false,\n "isIgnoreOKColors": false,\n "links": [ ],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "aggregation": "Last",\n "alias": "All",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ceph_osd_metadata{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "All",\n "refId": "A",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "In",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_osd_in{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "In",\n "refId": "B",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "Out",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Warning / Critical",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_osd_in{cluster=~\\"$cluster\\", } == bool 0)",\n "format": "time_series",\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Out",\n "refId": "C",\n "units": "none",\n "valueHandler": "Number Threshold",\n "warn": 1\n },\n {\n "aggregation": "Last",\n "alias": "Up",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_osd_up{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Up",\n "refId": "D",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "Down",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Warning / Critical",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_osd_up{cluster=~\\"$cluster\\", } == bool 0)",\n "format": "time_series",\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Down",\n "refId": "E",\n "units": "none",\n "valueHandler": "Number Threshold",\n "warn": 1\n }\n ],\n "title": "OSDs",\n "transparent": false,\n "type": "stat"\n },\n {\n "colorMode": "Panel",\n "colors": {\n "crit": "rgba(245, 54, 54, 0.9)",\n "disable": "rgba(128, 128, 128, 0.9)",\n "ok": "rgba(50, 128, 45, 0.9)",\n "warn": "rgba(237, 129, 40, 0.9)"\n },\n "cornerRadius": 1,\n "datasource": "$datasource",\n "description": "",\n "displayName": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "none"\n }\n },\n "flipCard": false,\n "flipTime": 5,\n "fontFormat": "Regular",\n "gridPos": {\n "h": 6,\n "w": 3,\n "x": 21,\n "y": 1\n },\n "id": 9,\n "isAutoScrollOnOverflow": false,\n "isGrayOnNoData": false,\n "isHideAlertsOnDisable": false,\n "isIgnoreOKColors": false,\n "links": [ ],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "aggregation": "Last",\n "alias": "Active",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ceph_mgr_status{cluster=~\\"$cluster\\", } == 1) or vector(0)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Active",\n "refId": "A",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "Standby",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ceph_mgr_status{cluster=~\\"$cluster\\", } == 0) or vector(0)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Standby",\n "refId": "B",\n "units": "none",\n "valueHandler": "Number Threshold"\n }\n ],\n "title": "MGRs",\n "transparent": false,\n "type": "stat"\n },\n {\n "colorMode": "Panel",\n "colors": {\n "crit": "rgba(245, 54, 54, 0.9)",\n "disable": "rgba(128, 128, 128, 0.9)",\n "ok": "rgba(50, 128, 45, 0.9)",\n "warn": "rgba(237, 129, 40, 0.9)"\n },\n "cornerRadius": 1,\n "datasource": "$datasource",\n "description": "",\n "displayName": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 1\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "Critical"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "red",\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Warning"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "#987d24",\n "mode": "fixed"\n }\n }\n ]\n }\n ]\n },\n "flipCard": false,\n "flipTime": 5,\n "fontFormat": "Regular",\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 0,\n "y": 4\n },\n "id": 10,\n "isAutoScrollOnOverflow": false,\n "isGrayOnNoData": false,\n "isHideAlertsOnDisable": false,\n "isIgnoreOKColors": false,\n "links": [ ],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "aggregation": "Last",\n "alias": "Active",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ALERTS{alertstate=\\"firing\\",alertname=~\\"^Ceph.+\\", severity=\\"critical\\", cluster=~\\"$cluster\\", }) OR vector(0)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Critical",\n "refId": "A",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "Standby",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ALERTS{alertstate=\\"firing\\",alertname=~\\"^Ceph.+\\", severity=\\"warning\\", cluster=~\\"$cluster\\", }) OR vector(0)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Warning",\n "refId": "B",\n "units": "none",\n "valueHandler": "Number Threshold"\n }\n ],\n "title": "Firing Alerts",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "displayName": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(50, 172, 45, 0.97)",\n "value": null\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 0.025000000000000001\n },\n {\n "color": "rgba(245, 54, 54, 0.9)",\n "value": 0.10000000000000001\n }\n ]\n },\n "unit": "bytes"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 6,\n "y": 4\n },\n "id": 11,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "ceph_cluster_total_used_bytes{cluster=~\\"$cluster\\", }",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "Used Capacity",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "displayName": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "ops"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 9,\n "y": 4\n },\n "id": 12,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(irate(ceph_osd_op_w{cluster=~\\"$cluster\\", }[1m]))",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "Write IOPS",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "displayName": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "#d44a3a",\n "value": null\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 0\n },\n {\n "color": "#9ac48a",\n "value": 0\n }\n ]\n },\n "unit": "ops"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 12,\n "y": 4\n },\n "id": 13,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(irate(ceph_osd_op_r{cluster=~\\"$cluster\\", }[1m]))",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "Read IOPS",\n "transparent": false,\n "type": "stat"\n },\n {\n "colorMode": "Panel",\n "colors": {\n "crit": "rgba(245, 54, 54, 0.9)",\n "disable": "rgba(128, 128, 128, 0.9)",\n "ok": "rgba(50, 128, 45, 0.9)",\n "warn": "rgba(237, 129, 40, 0.9)"\n },\n "cornerRadius": 1,\n "datasource": "$datasource",\n "description": "",\n "displayName": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "none"\n }\n },\n "flipCard": false,\n "flipTime": 5,\n "fontFormat": "Regular",\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 15,\n "y": 4\n },\n "id": 14,\n "isAutoScrollOnOverflow": false,\n "isGrayOnNoData": false,\n "isHideAlertsOnDisable": false,\n "isIgnoreOKColors": false,\n "links": [ ],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "aggregation": "Last",\n "alias": "In Quorum",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_mon_quorum_status{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "In Quorum",\n "refId": "A",\n "units": "none",\n "valueHandler": "Text Only"\n },\n {\n "aggregation": "Last",\n "alias": "Total",\n "crit": 1,\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ceph_mon_quorum_status{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Total",\n "refId": "B",\n "units": "none",\n "valueHandler": "Text Only",\n "warn": 2\n },\n {\n "aggregation": "Last",\n "alias": "MONs out of Quorum",\n "crit": 1.6000000000000001,\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Warning / Critical",\n "displayType": "Annotation",\n "displayValueWithAlias": "Never",\n "expr": "count(ceph_mon_quorum_status{cluster=~\\"$cluster\\", }) - sum(ceph_mon_quorum_status{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "MONs out of Quorum",\n "range": true,\n "refId": "C",\n "units": "none",\n "valueHandler": "Number Threshold",\n "warn": 1.1000000000000001\n }\n ],\n "title": "Monitors",\n "transparent": false,\n "type": "stat"\n },\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 7\n },\n "id": 15,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "CLUSTER STATS",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "datasource": {\n "type": "datasource",\n "uid": "grafana"\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 0,\n "y": 8\n },\n "id": 16,\n "limit": 10,\n "onlyAlertsOnDashboard": true,\n "options": {\n "alertInstanceLabelFilter": "{alertname=~\\"^Ceph.+\\", cluster=~\\"$cluster\\", }",\n "alertName": "",\n "dashboardAlerts": false,\n "groupBy": [ ],\n "groupMode": "default",\n "maxItems": 20,\n "sortOrder": 1,\n "stateFilter": {\n "error": true,\n "firing": true,\n "noData": false,\n "normal": false,\n "pending": true\n },\n "viewMode": "list"\n },\n "show": "current",\n "sortOrder": 1,\n "stateFilter": [ ],\n "title": "Alerts",\n "type": "alertlist"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 40,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 0,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "percentage",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "#c0921f",\n "value": 75\n },\n {\n "color": "#E02F44",\n "value": 85\n }\n ]\n },\n "unit": "bytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "Total Capacity"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "red",\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Used"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "green",\n "mode": "fixed"\n }\n },\n {\n "id": "custom.thresholdsStyle",\n "value": {\n "mode": "dashed"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 8,\n "y": 8\n },\n "id": 17,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [\n "last"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "ceph_cluster_total_bytes{cluster=~\\"$cluster\\", }",\n "format": "time_series",\n "instant": false,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Total Capacity",\n "range": true,\n "refId": "A",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "ceph_cluster_total_used_bytes{cluster=~\\"$cluster\\", }",\n "format": "time_series",\n "instant": false,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Used",\n "range": true,\n "refId": "B",\n "step": 300\n }\n ],\n "title": "Capacity",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 85\n }\n ]\n },\n "unit": "bytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 16,\n "y": 8\n },\n "id": 18,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [\n "mean",\n "lastNotNull",\n "max",\n "min"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(irate(ceph_osd_op_w_in_bytes{cluster=~\\"$cluster\\", }[5m]))",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Write",\n "range": true,\n "refId": "A",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "sum(irate(ceph_osd_op_r_out_bytes{cluster=~\\"$cluster\\", }[5m]))",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Read",\n "range": true,\n "refId": "B",\n "step": 300\n }\n ],\n "title": "Cluster Throughput",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "bytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 0,\n "y": 16\n },\n "id": 19,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [\n "mean",\n "lastNotNull",\n "max",\n "min"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(irate(ceph_osd_op_w{cluster=~\\"$cluster\\", }[1m]))",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Write",\n "range": true,\n "refId": "A",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "sum(irate(ceph_osd_op_r{cluster=~\\"$cluster\\", }[1m]))",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Read",\n "range": true,\n "refId": "B",\n "step": 300\n }\n ],\n "title": "IOPS",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "bytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 8,\n "y": 16\n },\n "id": 20,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "right",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "(ceph_pool_bytes_used{cluster=~\\"$cluster\\", }) *on (pool_id) group_left(name)(ceph_pool_metadata{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Pool Used Bytes",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "bytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "rbd Stored"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "transparent",\n "mode": "fixed"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 16,\n "y": 16\n },\n "id": 21,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "table",\n "placement": "right",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "(ceph_pool_stored_raw{cluster=~\\"$cluster\\", }) *on (pool_id) group_left(name)(ceph_pool_metadata{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "hide": false,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Pool Used RAW Bytes",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 0,\n "y": 24\n },\n "id": 22,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "(ceph_pool_quota_objects{cluster=~\\"$cluster\\", }) *on (pool_id) group_left(name)(ceph_pool_metadata{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Pool Objects Quota",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "bytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 8,\n "y": 24\n },\n "id": 23,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "(ceph_pool_quota_bytes{cluster=~\\"$cluster\\", }) *on (pool_id) group_left(name)(ceph_pool_metadata{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Pool Quota Bytes",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 16,\n "y": 24\n },\n "id": 24,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "right",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "(ceph_pool_objects{cluster=~\\"$cluster\\", }) * on (pool_id) group_left(name)(ceph_pool_metadata{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "refId": "A"\n }\n ],\n "title": "Objects Per Pool",\n "type": "timeseries"\n },\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 31\n },\n "id": 25,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "OBJECTS",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 2,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byRegexp",\n "options": "/^Total.*$/"\n },\n "properties": [\n {\n "id": "custom.stacking",\n "value": {\n "group": false,\n "mode": "normal"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 12,\n "w": 6,\n "x": 0,\n "y": 32\n },\n "id": 26,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": false\n },\n "tooltip": {\n "mode": "multi",\n "sort": "asc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pool_objects{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Total",\n "range": true,\n "refId": "A",\n "step": 200\n }\n ],\n "title": "OSD Type Count",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 2,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byRegexp",\n "options": "/^Total.*$/"\n },\n "properties": [\n {\n "id": "custom.stacking",\n "value": {\n "group": false,\n "mode": "normal"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 12,\n "w": 8,\n "x": 6,\n "y": 32\n },\n "id": 27,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "right",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "asc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_active{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Active",\n "range": true,\n "refId": "A"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_clean{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Clean",\n "range": true,\n "refId": "B"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_peering{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Peering",\n "range": true,\n "refId": "C"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_degraded{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Degraded",\n "range": true,\n "refId": "D",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_stale{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Stale",\n "range": true,\n "refId": "E",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_unclean_pgs{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Unclean",\n "range": true,\n "refId": "F",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_undersized{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Undersized",\n "range": true,\n "refId": "G",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_incomplete{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Incomplete",\n "range": true,\n "refId": "H"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_forced_backfill{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Forced Backfill",\n "range": true,\n "refId": "I"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_forced_recovery{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Forced Recovery",\n "range": true,\n "refId": "J"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_creating{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Creating",\n "range": true,\n "refId": "K"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_wait_backfill{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Wait Backfill",\n "range": true,\n "refId": "L"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_deep{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Deep",\n "range": true,\n "refId": "M"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_scrubbing{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Scrubbing",\n "range": true,\n "refId": "N"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_recovering{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Recovering",\n "range": true,\n "refId": "O"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_repair{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Repair",\n "range": true,\n "refId": "P"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_down{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Down",\n "range": true,\n "refId": "Q"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_peered{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Peered",\n "range": true,\n "refId": "R"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_backfill{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Backfill",\n "range": true,\n "refId": "S"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_remapped{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Remapped",\n "range": true,\n "refId": "T"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_backfill_toofull{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Backfill Toofull",\n "range": true,\n "refId": "U"\n }\n ],\n "title": "PGs State",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 2,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byRegexp",\n "options": "/^Total.*$/"\n },\n "properties": [\n {\n "id": "custom.stacking",\n "value": {\n "group": false,\n "mode": "normal"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 6,\n "w": 10,\n "x": 14,\n "y": 32\n },\n "id": 28,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [\n "mean",\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "right",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "asc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_degraded{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Degraded",\n "range": true,\n "refId": "A",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_stale{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Stale",\n "range": true,\n "refId": "B",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_undersized{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Undersized",\n "range": true,\n "refId": "C",\n "step": 300\n }\n ],\n "title": "Stuck PGs",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 2,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 6,\n "w": 10,\n "x": 14,\n "y": 38\n },\n "id": 29,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": false\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(irate(ceph_osd_recovery_ops{cluster=~\\"$cluster\\", }[$interval]))",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "OPS",\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Recovery Operations",\n "type": "timeseries"\n },\n {\n "collapse": false,\n "collapsed": true,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 44\n },\n "id": 30,\n "panels": [\n {\n "cards": {\n "cardPadding": null,\n "cardRound": null\n },\n "color": {\n "cardColor": "#b4ff00",\n "colorScale": "sqrt",\n "colorScheme": "interpolateOranges",\n "exponent": 0.5,\n "mode": "opacity"\n },\n "dataFormat": "timeseries",\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "scaleDistribution": {\n "type": "linear"\n }\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 0,\n "y": 42\n },\n "heatmap": { },\n "hideZeroBuckets": false,\n "highlightCards": true,\n "id": 31,\n "legend": {\n "show": true\n },\n "options": {\n "calculate": true,\n "calculation": {\n "yBuckets": {\n "mode": "count",\n "scale": {\n "log": 2,\n "type": "log"\n },\n "value": "1"\n }\n },\n "cellGap": 2,\n "cellValues": { },\n "color": {\n "exponent": 0.5,\n "fill": "#b4ff00",\n "mode": "opacity",\n "reverse": false,\n "scale": "exponential",\n "scheme": "Oranges",\n "steps": 128\n },\n "exemplars": {\n "color": "rgba(255,0,255,0.7)"\n },\n "filterValues": {\n "le": 1.0000000000000001e-09\n },\n "legend": {\n "show": true\n },\n "rowsFrame": {\n "layout": "auto"\n },\n "showValue": "never",\n "tooltip": {\n "show": true,\n "yHistogram": false\n },\n "yAxis": {\n "axisPlacement": "left",\n "min": "0",\n "reverse": false,\n "unit": "ms"\n }\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "ceph_osd_apply_latency_ms{cluster=~\\"$cluster\\", }",\n "format": "time_series",\n "instant": false,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "OSD Apply Latency Distribution",\n "tooltip": {\n "show": true,\n "showHistogram": false\n },\n "type": "heatmap",\n "xAxis": {\n "show": true\n },\n "xBucketNumber": null,\n "xBucketSize": "",\n "yAxis": {\n "decimals": null,\n "format": "ms",\n "logBase": 2,\n "max": null,\n "min": "0",\n "show": true,\n "splitFactor": 1\n },\n "yBucketBound": "auto",\n "yBucketNumber": null,\n "yBucketSize": 10\n },\n {\n "cards": {\n "cardPadding": null,\n "cardRound": null\n },\n "color": {\n "cardColor": "#65c5db",\n "colorScale": "sqrt",\n "colorScheme": "interpolateOranges",\n "exponent": 0.5,\n "mode": "opacity"\n },\n "dataFormat": "timeseries",\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "scaleDistribution": {\n "type": "linear"\n }\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 12,\n "y": 42\n },\n "heatmap": { },\n "hideZeroBuckets": false,\n "highlightCards": true,\n "id": 32,\n "legend": {\n "show": true\n },\n "options": {\n "calculate": true,\n "calculation": {\n "yBuckets": {\n "mode": "count",\n "scale": {\n "log": 2,\n "type": "log"\n }\n }\n },\n "cellGap": 2,\n "cellValues": { },\n "color": {\n "exponent": 0.5,\n "fill": "#65c5db",\n "mode": "opacity",\n "reverse": false,\n "scale": "exponential",\n "scheme": "Oranges",\n "steps": 128\n },\n "exemplars": {\n "color": "rgba(255,0,255,0.7)"\n },\n "filterValues": {\n "le": 1.0000000000000001e-09\n },\n "legend": {\n "show": true\n },\n "rowsFrame": {\n "layout": "auto"\n },\n "showValue": "never",\n "tooltip": {\n "show": true,\n "yHistogram": false\n },\n "yAxis": {\n "axisPlacement": "left",\n "min": "0",\n "reverse": false,\n "unit": "ms"\n }\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "ceph_osd_commit_latency_ms{cluster=~\\"$cluster\\", }",\n "format": "time_series",\n "instant": false,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "OSD Commit Latency Distribution",\n "tooltip": {\n "show": true,\n "showHistogram": false\n },\n "type": "heatmap",\n "xAxis": {\n "show": true\n },\n "xBucketNumber": null,\n "xBucketSize": "",\n "yAxis": {\n "decimals": null,\n "format": "ms",\n "logBase": 2,\n "max": null,\n "min": "0",\n "show": true,\n "splitFactor": 1\n },\n "yBucketBound": "auto",\n "yBucketNumber": null,\n "yBucketSize": 10\n },\n {\n "cards": {\n "cardPadding": null,\n "cardRound": null\n },\n "color": {\n "cardColor": "#806eb7",\n "colorScale": "sqrt",\n "colorScheme": "interpolateOranges",\n "exponent": 0.5,\n "mode": "opacity"\n },\n "dataFormat": "timeseries",\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "scaleDistribution": {\n "type": "linear"\n }\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 0,\n "y": 50\n },\n "heatmap": { },\n "hideZeroBuckets": false,\n "highlightCards": true,\n "id": 33,\n "legend": {\n "show": true\n },\n "options": {\n "calculate": true,\n "calculation": {\n "yBuckets": {\n "mode": "count",\n "scale": {\n "log": 2,\n "type": "log"\n }\n }\n },\n "cellGap": 2,\n "cellValues": { },\n "color": {\n "exponent": 0.5,\n "fill": "#806eb7",\n "mode": "opacity",\n "reverse": false,\n "scale": "exponential",\n "scheme": "Oranges",\n "steps": 128\n },\n "exemplars": {\n "color": "rgba(255,0,255,0.7)"\n },\n "filterValues": {\n "le": 1.0000000000000001e-09\n },\n "legend": {\n "show": true\n },\n "rowsFrame": {\n "layout": "auto"\n },\n "showValue": "never",\n "tooltip": {\n "show": true,\n "yHistogram": false\n },\n "yAxis": {\n "axisPlacement": "left",\n "decimals": 2,\n "min": "0",\n "reverse": false,\n "unit": "ms"\n }\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "rate(ceph_osd_op_r_latency_sum{cluster=~\\"$cluster\\", }[5m]) / rate(ceph_osd_op_r_latency_count{cluster=~\\"$cluster\\", }[5m]) >= 0",\n "format": "time_series",\n "instant": false,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "OSD Read Op Latency Distribution",\n "tooltip": {\n "show": true,\n "showHistogram": false\n },\n "type": "heatmap",\n "xAxis": {\n "show": true\n },\n "xBucketNumber": null,\n "xBucketSize": "",\n "yAxis": {\n "decimals": null,\n "format": "ms",\n "logBase": 2,\n "max": null,\n "min": "0",\n "show": true,\n "splitFactor": 1\n },\n "yBucketBound": "auto",\n "yBucketNumber": null,\n "yBucketSize": null\n },\n {\n "cards": {\n "cardPadding": null,\n "cardRound": null\n },\n "color": {\n "cardColor": "#f9934e",\n "colorScale": "sqrt",\n "colorScheme": "interpolateOranges",\n "exponent": 0.5,\n "mode": "opacity"\n },\n "dataFormat": "timeseries",\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "scaleDistribution": {\n "type": "linear"\n }\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 12,\n "y": 50\n },\n "heatmap": { },\n "hideZeroBuckets": false,\n "highlightCards": true,\n "id": 34,\n "legend": {\n "show": true\n },\n "options": {\n "calculate": true,\n "calculation": {\n "yBuckets": {\n "mode": "count",\n "scale": {\n "log": 2,\n "type": "log"\n }\n }\n },\n "cellGap": 2,\n "cellValues": { },\n "color": {\n "exponent": 0.5,\n "fill": "#f9934e",\n "mode": "opacity",\n "reverse": false,\n "scale": "exponential",\n "scheme": "Oranges",\n "steps": 128\n },\n "exemplars": {\n "color": "rgba(255,0,255,0.7)"\n },\n "filterValues": {\n "le": 1.0000000000000001e-09\n },\n "legend": {\n "show": true\n },\n "rowsFrame": {\n "layout": "auto"\n },\n "showValue": "never",\n "tooltip": {\n "show": true,\n "yHistogram": false\n },\n "yAxis": {\n "axisPlacement": "left",\n "decimals": 2,\n "min": "0",\n "reverse": false,\n "unit": "ms"\n }\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "rate(ceph_osd_op_w_latency_sum{cluster=~\\"$cluster\\", }[5m]) / rate(ceph_osd_op_w_latency_count{cluster=~\\"$cluster\\", }[5m]) >= 0",\n "format": "time_series",\n "instant": false,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "OSD Write Op Latency Distribution",\n "tooltip": {\n "show": true,\n "showHistogram": false\n },\n "type": "heatmap",\n "xAxis": {\n "show": true\n },\n "xBucketNumber": null,\n "xBucketSize": "",\n "yAxis": {\n "decimals": null,\n "format": "ms",\n "logBase": 2,\n "max": null,\n "min": "0",\n "show": true,\n "splitFactor": 1\n },\n "yBucketBound": "auto",\n "yBucketNumber": null,\n "yBucketSize": null\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "ms"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 12,\n "x": 0,\n "y": 58\n },\n "id": 35,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "avg(rate(ceph_osd_op_r_latency_sum{cluster=~\\"$cluster\\", }[5m]) / rate(ceph_osd_op_r_latency_count{cluster=~\\"$cluster\\", }[5m]) >= 0)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Read",\n "refId": "A"\n },\n {\n "datasource": "$datasource",\n "expr": "avg(rate(ceph_osd_op_w_latency_sum{cluster=~\\"$cluster\\", }[5m]) / rate(ceph_osd_op_w_latency_count{cluster=~\\"$cluster\\", }[5m]) >= 0)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Write",\n "refId": "B"\n }\n ],\n "title": "Recovery Operations",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "ms"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 12,\n "x": 12,\n "y": 58\n },\n "id": 36,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull",\n "max"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "avg(ceph_osd_apply_latency_ms{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "apply",\n "metric": "ceph_osd_perf_apply_latency_seconds",\n "refId": "A",\n "step": 4\n },\n {\n "datasource": "$datasource",\n "expr": "avg(ceph_osd_commit_latency_ms{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "commit",\n "metric": "ceph_osd_perf_commit_latency_seconds",\n "refId": "B",\n "step": 4\n }\n ],\n "title": "AVG OSD Apply + Commit Latency",\n "type": "timeseries"\n }\n ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "LATENCY",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "collapse": true,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 45\n },\n "id": 37,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "columns": [ ],\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "left",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": false,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "Time"\n },\n "properties": [\n {\n "id": "custom.hidden",\n "value": true\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 6,\n "w": 24,\n "x": 0,\n "y": 46\n },\n "id": 38,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "9.4.7",\n "styles": "",\n "targets": [\n {\n "datasource": "$datasource",\n "exemplar": false,\n "expr": "count by (ceph_version)(ceph_osd_metadata{cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "OSD Services",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": "$datasource",\n "exemplar": false,\n "expr": "count by (ceph_version)(ceph_mon_metadata{cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Mon Services",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": "$datasource",\n "exemplar": false,\n "expr": "count by (ceph_version)(ceph_mds_metadata{cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "MDS Services",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": "$datasource",\n "exemplar": false,\n "expr": "count by (ceph_version)(ceph_rgw_metadata{cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "RGW Services",\n "range": false,\n "refId": "D"\n },\n {\n "datasource": "$datasource",\n "exemplar": false,\n "expr": "count by (ceph_version)(ceph_mgr_metadata{cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "MGR Services",\n "range": false,\n "refId": "E"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Ceph Versions",\n "transformations": [\n {\n "id": "merge",\n "options": { }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": { },\n "indexByName": { },\n "renameByName": {\n "Time": "",\n "Value #A": "OSD Services",\n "Value #B": "Mon Services",\n "Value #C": "MDS Services",\n "Value #D": "RGW Services",\n "Value #E": "MGR Services",\n "ceph_version": "Ceph Version"\n }\n }\n }\n ],\n "type": "table"\n }\n ],\n "refresh": "1m",\n "rows": [ ],\n "schemaVersion": 38,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "auto": true,\n "auto_count": 10,\n "auto_min": "1m",\n "current": {\n "text": "$__auto_interval_interval",\n "value": "$__auto_interval_interval"\n },\n "hide": 0,\n "label": "Interval",\n "name": "interval",\n "options": [\n {\n "selected": true,\n "text": "auto",\n "value": "$__auto_interval_interval"\n },\n {\n "selected": false,\n "text": "5s",\n "value": "5s"\n },\n {\n "selected": false,\n "text": "10s",\n "value": "10s"\n },\n {\n "selected": false,\n "text": "30s",\n "value": "30s"\n },\n {\n "selected": false,\n "text": "1m",\n "value": "1m"\n },\n {\n "selected": false,\n "text": "10m",\n "value": "10m"\n },\n {\n "selected": false,\n "text": "30m",\n "value": "30m"\n },\n {\n "selected": false,\n "text": "1h",\n "value": "1h"\n },\n {\n "selected": false,\n "text": "6h",\n "value": "6h"\n },\n {\n "selected": false,\n "text": "12h",\n "value": "12h"\n },\n {\n "selected": false,\n "text": "1d",\n "value": "1d"\n },\n {\n "selected": false,\n "text": "7d",\n "value": "7d"\n },\n {\n "selected": false,\n "text": "14d",\n "value": "14d"\n },\n {\n "selected": false,\n "text": "30d",\n "value": "30d"\n }\n ],\n "query": "5s,10s,30s,1m,10m,30m,1h,6h,12h,1d,7d,14d,30d",\n "refresh": 2,\n "type": "interval",\n "valuelabels": { }\n }\n ]\n },\n "time": {\n "from": "now-6h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "Ceph Cluster - Advanced",\n "uid": "dn13KBeTv",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/osds-overview.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.0.0"\n },\n {\n "id": "grafana-piechart-panel",\n "name": "Pie Chart",\n "type": "panel",\n "version": "1.3.3"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n },\n {\n "id": "table",\n "name": "Table",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "ms"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "avg(\\n rate(ceph_osd_op_r_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_r_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval]) * 1000\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "AVG read",\n "refId": "A"\n },\n {\n "expr": "max(\\n rate(ceph_osd_op_r_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_r_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval]) * 1000\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "MAX read",\n "refId": "B"\n },\n {\n "expr": "quantile(0.95,\\n (\\n rate(ceph_osd_op_r_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_r_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval]) * 1000\\n )\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "@95%ile",\n "refId": "C"\n }\n ],\n "title": "OSD Read Latencies",\n "type": "timeseries"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "null",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "ceph_daemon"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "OSD ID"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Latency (ms)"\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 4,\n "x": 8,\n "y": 0\n },\n "id": 3,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "10.4.0",\n "styles": "",\n "targets": [\n {\n "expr": "topk(10,\\n (sort(\\n (\\n rate(ceph_osd_op_r_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_r_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n 1000\\n )\\n ))\\n)\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Highest READ Latencies",\n "transformations": [\n {\n "id": "merge",\n "options": {\n "reducers": [ ]\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time": true,\n "cluster": true\n },\n "includeByName": { },\n "indexByName": { },\n "renameByName": { }\n }\n }\n ],\n "type": "table"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "ms"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 12,\n "y": 0\n },\n "id": 4,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "avg(\\n rate(ceph_osd_op_w_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_w_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval]) * 1000\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "AVG write",\n "refId": "A"\n },\n {\n "expr": "max(\\n rate(ceph_osd_op_w_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_w_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval]) * 1000\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "MAX write",\n "refId": "B"\n },\n {\n "expr": "quantile(0.95,\\n (\\n rate(ceph_osd_op_w_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_w_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval]) * 1000\\n )\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "@95%ile write",\n "refId": "C"\n }\n ],\n "title": "OSD Write Latencies",\n "type": "timeseries"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "description": "This table shows the osd\'s that are delivering the 10 highest write latencies within the cluster",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "null",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "ceph_daemon"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "OSD ID"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Latency (ms)"\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "mappings",\n "value": [\n {\n "options": {\n "NaN": {\n "index": 0,\n "text": "0.00"\n }\n },\n "type": "value"\n }\n ]\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 4,\n "x": 20,\n "y": 0\n },\n "id": 5,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "10.4.0",\n "styles": "",\n "targets": [\n {\n "expr": "topk(10,\\n (sort(\\n (rate(ceph_osd_op_w_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_w_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n 1000)\\n ))\\n)\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Highest WRITE Latencies",\n "transformations": [\n {\n "id": "merge",\n "options": {\n "reducers": [ ]\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time": true,\n "cluster": true\n },\n "includeByName": { },\n "indexByName": { },\n "renameByName": { }\n }\n }\n ],\n "type": "table"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n }\n },\n "mappings": [ ]\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 4,\n "x": 0,\n "y": 8\n },\n "id": 6,\n "options": {\n "displayLabels": [\n "percent"\n ],\n "legend": {\n "calcs": [ ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "values": [\n "percent",\n "value"\n ]\n },\n "pieType": "pie",\n "reduceOptions": { },\n "tooltip": {\n "mode": "single",\n "sort": "none"\n }\n },\n "targets": [\n {\n "expr": "count by (device_class) (ceph_osd_metadata{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device_class}}",\n "refId": "A"\n }\n ],\n "title": "OSD Types Summary",\n "type": "piechart"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n }\n },\n "mappings": [ ]\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 4,\n "x": 4,\n "y": 8\n },\n "id": 7,\n "options": {\n "displayLabels": [\n "percent"\n ],\n "legend": {\n "calcs": [ ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "values": [\n "percent",\n "value"\n ]\n },\n "pieType": "pie",\n "reduceOptions": { },\n "tooltip": {\n "mode": "single",\n "sort": "none"\n }\n },\n "targets": [\n {\n "expr": "count(ceph_bluefs_wal_total_bytes{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "bluestore",\n "refId": "A"\n },\n {\n "expr": "absent(ceph_bluefs_wal_total_bytes{cluster=~\\"$cluster\\", }) * count(ceph_osd_metadata{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "filestore",\n "refId": "B"\n }\n ],\n "title": "OSD Objectstore Types",\n "type": "piechart"\n },\n {\n "datasource": "$datasource",\n "description": "The pie chart shows the various OSD sizes used within the cluster",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n }\n },\n "mappings": [ ]\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 4,\n "x": 8,\n "y": 8\n },\n "id": 8,\n "options": {\n "displayLabels": [\n "percent"\n ],\n "legend": {\n "calcs": [ ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "values": [\n "percent",\n "value"\n ]\n },\n "pieType": "pie",\n "reduceOptions": { },\n "tooltip": {\n "mode": "single",\n "sort": "none"\n }\n },\n "targets": [\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster\\", } < 1099511627776)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<1TB",\n "refId": "A"\n },\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster\\", } >= 1099511627776 < 2199023255552)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<2TB",\n "refId": "B"\n },\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster\\", } >= 2199023255552 < 3298534883328)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<3TB",\n "refId": "C"\n },\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster\\", } >= 3298534883328 < 4398046511104)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<4TB",\n "refId": "D"\n },\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster\\", } >= 4398046511104 < 6597069766656)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<6TB",\n "refId": "E"\n },\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster\\", } >= 6597069766656 < 8796093022208)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<8TB",\n "refId": "F"\n },\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster\\", } >= 8796093022208 < 10995116277760)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<10TB",\n "refId": "G"\n },\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster\\", } >= 10995116277760 < 13194139533312)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<12TB",\n "refId": "H"\n },\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster\\", } >= 13194139533312)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<12TB+",\n "refId": "I"\n }\n ],\n "title": "OSD Size Summary",\n "type": "piechart"\n },\n {\n "aliasColors": { },\n "bars": true,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 12,\n "y": 8\n },\n "id": 9,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "ceph_osd_numpg{cluster=~\\"$cluster\\", }",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "PGs per OSD",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Distribution of PGs per OSD",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": 20,\n "mode": "histogram",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": "# of OSDs",\n "logBase": 1,\n "max": null,\n "min": "0",\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": "0",\n "show": true\n }\n ]\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": true,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "This gauge panel shows onode Hits ratio to help determine if increasing RAM per OSD could help improve the performance of the cluster",\n "format": "percentunit",\n "gauge": {\n "maxValue": 1,\n "minValue": 0,\n "show": true,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 8,\n "w": 4,\n "x": 20,\n "y": 8\n },\n "id": 10,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum(ceph_bluestore_onode_hits{cluster=~\\"$cluster\\", }) / (\\n sum(ceph_bluestore_onode_hits{cluster=~\\"$cluster\\", }) +\\n sum(ceph_bluestore_onode_misses{cluster=~\\"$cluster\\", })\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": ".75",\n "title": "OSD onode Hits Ratio",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 16\n },\n "id": 11,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "R/W Profile",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 24,\n "x": 0,\n "y": 17\n },\n "id": 12,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "round(sum(rate(ceph_pool_rd{cluster=~\\"$cluster\\", }[$__rate_interval])))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Reads",\n "refId": "A"\n },\n {\n "expr": "round(sum(rate(ceph_pool_wr{cluster=~\\"$cluster\\", }[$__rate_interval])))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Writes",\n "refId": "B"\n }\n ],\n "title": "Read/Write Profile",\n "type": "timeseries"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "description": "This table shows the 10 OSDs with the highest number of slow ops",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "null",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "ceph_daemon"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "OSD ID"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Slow Ops"\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 5,\n "x": 0,\n "y": 25\n },\n "id": 13,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "10.4.0",\n "styles": "",\n "targets": [\n {\n "expr": "topk(10,\\n (ceph_daemon_health_metrics{type=\\"SLOW_OPS\\", ceph_daemon=~\\"osd.*\\"})\\n)\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Top Slow Ops",\n "transformations": [\n {\n "id": "merge",\n "options": {\n "reducers": [ ]\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time": true,\n "__name__": true,\n "cluster": true,\n "instance": true,\n "job": true,\n "type": true\n },\n "includeByName": { },\n "indexByName": { },\n "renameByName": { }\n }\n }\n ],\n "type": "table"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "OSD Overview",\n "uid": "lo02I1Aiz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/rbd-overview.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.4.2"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n },\n {\n "id": "prometheus",\n "name": "Prometheus",\n "type": "datasource",\n "version": "5.0.0"\n },\n {\n "id": "table",\n "name": "Table",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "round(sum(rate(ceph_rbd_write_ops{cluster=~\\"$cluster\\", }[$__rate_interval])))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Writes",\n "refId": "A"\n },\n {\n "expr": "round(sum(rate(ceph_rbd_read_ops{cluster=~\\"$cluster\\", }[$__rate_interval])))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Reads",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "IOPS",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "Bps"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 8,\n "y": 0\n },\n "id": 3,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "round(sum(rate(ceph_rbd_write_bytes{cluster=~\\"$cluster\\", }[$__rate_interval])))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Write",\n "refId": "A"\n },\n {\n "expr": "round(sum(rate(ceph_rbd_read_bytes{cluster=~\\"$cluster\\", }[$__rate_interval])))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Read",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Throughput",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "Bps",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "ns"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 16,\n "y": 0\n },\n "id": 4,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "round(\\n sum(rate(ceph_rbd_write_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval])) /\\n sum(rate(ceph_rbd_write_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval]))\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Write",\n "refId": "A"\n },\n {\n "expr": "round(\\n sum(rate(ceph_rbd_read_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval])) /\\n sum(rate(ceph_rbd_read_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval]))\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Read",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Average Latency",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "ns",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "null",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "pool"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Pool"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "image"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Image"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "IOPS"\n },\n {\n "id": "unit",\n "value": "iops"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 0,\n "y": 7\n },\n "id": 5,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "10.4.0",\n "styles": "",\n "targets": [\n {\n "expr": "topk(10,\\n (\\n sort((\\n rate(ceph_rbd_write_ops{cluster=~\\"$cluster\\", }[$__rate_interval]) +\\n on (image, pool, namespace) rate(ceph_rbd_read_ops{cluster=~\\"$cluster\\", }[$__rate_interval])\\n ))\\n )\\n)\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Highest IOPS",\n "transformations": [\n {\n "id": "merge",\n "options": {\n "reducers": [ ]\n }\n }\n ],\n "type": "table"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "null",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "pool"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Pool"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "image"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Image"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Throughput"\n },\n {\n "id": "unit",\n "value": "Bps"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 8,\n "y": 7\n },\n "id": 6,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "10.4.0",\n "styles": "",\n "targets": [\n {\n "expr": "topk(10,\\n sort(\\n sum(\\n rate(ceph_rbd_read_bytes{cluster=~\\"$cluster\\", }[$__rate_interval]) +\\n rate(ceph_rbd_write_bytes{cluster=~\\"$cluster\\", }[$__rate_interval])\\n ) by (pool, image, namespace)\\n )\\n)\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Highest Throughput",\n "transformations": [\n {\n "id": "merge",\n "options": {\n "reducers": [ ]\n }\n }\n ],\n "type": "table"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "null",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "pool"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Pool"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "image"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Image"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Latency"\n },\n {\n "id": "unit",\n "value": "ns"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 16,\n "y": 7\n },\n "id": 7,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "10.4.0",\n "styles": "",\n "targets": [\n {\n "expr": "topk(10,\\n sum(\\n rate(ceph_rbd_write_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n clamp_min(rate(ceph_rbd_write_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval]), 1) +\\n rate(ceph_rbd_read_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n clamp_min(rate(ceph_rbd_read_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval]), 1)\\n ) by (pool, image, namespace)\\n)\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Highest Latency",\n "transformations": [\n {\n "id": "merge",\n "options": {\n "reducers": [ ]\n }\n }\n ],\n "type": "table"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin",\n "overview"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "RBD Overview",\n "uid": "41FrpeUiz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/osd-device-details.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.3.2"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "OSD Performance",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "s"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 0,\n "y": 1\n },\n "id": 3,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "read",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "rate(ceph_osd_op_r_latency_sum{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_r_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval])\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "read",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_osd_op_w_latency_sum{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_w_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval])\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "write",\n "refId": "B"\n }\n ],\n "title": "$osd Latency",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 6,\n "y": 1\n },\n "id": 4,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "Reads",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "rate(ceph_osd_op_r{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", }[$__rate_interval])",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Reads",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_osd_op_w{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", }[$__rate_interval])",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Writes",\n "refId": "B"\n }\n ],\n "title": "$osd R/W IOPS",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "bytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 12,\n "y": 1\n },\n "id": 5,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "Read Bytes",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "rate(ceph_osd_op_r_out_bytes{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", }[$__rate_interval])",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Read Bytes",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_osd_op_w_in_bytes{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", }[$__rate_interval])",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Write Bytes",\n "refId": "B"\n }\n ],\n "title": "$osd R/W Bytes",\n "type": "timeseries"\n },\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 10\n },\n "id": 6,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "Physical Device Performance",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "s"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 0,\n "y": 11\n },\n "id": 7,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*Reads/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "(\\n label_replace(\\n rate(node_disk_read_time_seconds_total[$__rate_interval]) /\\n rate(node_disk_reads_completed_total[$__rate_interval]),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n ) and on (instance, device) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", },\\n \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n )\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{instance}}/{{device}} Reads",\n "refId": "A"\n },\n {\n "expr": "(\\n label_replace(\\n rate(node_disk_write_time_seconds_total[$__rate_interval]) /\\n rate(node_disk_writes_completed_total[$__rate_interval]),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\") and on (instance, device)\\n label_replace(\\n label_replace(\\n ceph_disk_occupation_human{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", }, \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n )\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{instance}}/{{device}} Writes",\n "refId": "B"\n }\n ],\n "title": "Physical Device Latency for $osd",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 6,\n "y": 11\n },\n "id": 8,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*Reads/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "label_replace(\\n rate(node_disk_writes_completed_total[$__rate_interval]),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) and on (instance, device) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", },\\n \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}} on {{instance}} Writes",\n "refId": "A"\n },\n {\n "expr": "label_replace(\\n rate(node_disk_reads_completed_total[$__rate_interval]),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) and on (instance, device) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", },\\n \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}} on {{instance}} Reads",\n "refId": "B"\n }\n ],\n "title": "Physical Device R/W IOPS for $osd",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "Bps"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 12,\n "y": 11\n },\n "id": 9,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*Reads/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "label_replace(\\n rate(node_disk_read_bytes_total[$__rate_interval]), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) and on (instance, device) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", },\\n \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{instance}} {{device}} Reads",\n "refId": "A"\n },\n {\n "expr": "label_replace(\\n rate(node_disk_written_bytes_total[$__rate_interval]), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) and on (instance, device) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", },\\n \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{instance}} {{device}} Writes",\n "refId": "B"\n }\n ],\n "title": "Physical Device R/W Bytes for $osd",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "percentunit"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 18,\n "y": 11\n },\n "id": 10,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "label_replace(\\n rate(node_disk_io_time_seconds_total[$__rate_interval]),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) and on (instance, device) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", }, \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}} on {{instance}}",\n "refId": "A"\n }\n ],\n "title": "Physical Device Util% for $osd",\n "type": "timeseries"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "OSD",\n "multi": false,\n "name": "osd",\n "options": [ ],\n "query": "label_values(ceph_osd_metadata{cluster=~\\"$cluster\\", }, ceph_daemon)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-3h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "OSD device details",\n "uid": "CrAHE0iZz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/cephfs-overview.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.3.2"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "MDS Performance",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Reads(-) / Writes (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "none"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 0,\n "y": 1\n },\n "id": 3,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*Reads/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "sum(rate(ceph_objecter_op_r{ceph_daemon=~\\"($mds_servers).*\\", cluster=~\\"$cluster\\", }[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Read Ops",\n "refId": "A"\n },\n {\n "expr": "sum(rate(ceph_objecter_op_w{ceph_daemon=~\\"($mds_servers).*\\", cluster=~\\"$cluster\\", }[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Write Ops",\n "refId": "B"\n }\n ],\n "title": "MDS Workload - $mds_servers",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Client Requests",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "none"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 12,\n "y": 1\n },\n "id": 4,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "ceph_mds_server_handle_client_request{ceph_daemon=~\\"($mds_servers).*\\", cluster=~\\"$cluster\\", }",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}}",\n "refId": "A"\n }\n ],\n "title": "Client Request Load - $mds_servers",\n "type": "timeseries"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": "MDS Server",\n "multi": false,\n "name": "mds_servers",\n "options": [ ],\n "query": "label_values(ceph_mds_inodes{cluster=~\\"$cluster\\", }, ceph_daemon)",\n "refresh": 1,\n "regex": "",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "MDS Performance",\n "uid": "tbO9LAiZz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/pool-detail.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.3.2"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n },\n {\n "id": "singlestat",\n "name": "Singlestat",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": true,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "format": "percentunit",\n "gauge": {\n "maxValue": 1,\n "minValue": 0,\n "show": true,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 7,\n "w": 7,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": true\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "(ceph_pool_stored{cluster=~\\"$cluster\\", } / (ceph_pool_stored{cluster=~\\"$cluster\\", } + ceph_pool_max_avail{cluster=~\\"$cluster\\", })) *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{name=~\\"$pool_name\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": ".7,.8",\n "title": "Capacity used",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": 100,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Time till pool is full assuming the average fill rate of the last 6 hours",\n "format": "s",\n "gauge": {\n "maxValue": false,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 7,\n "w": 5,\n "x": 7,\n "y": 0\n },\n "id": 3,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": ""\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "(ceph_pool_max_avail{cluster=~\\"$cluster\\", } / deriv(ceph_pool_stored{cluster=~\\"$cluster\\", }[6h])) *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{name=~\\"$pool_name\\", cluster=~\\"$cluster\\", } > 0\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "current",\n "title": "Time till full",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": false\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Objects out(-) / in(+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "ops"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 12,\n "x": 12,\n "y": 0\n },\n "id": 4,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "deriv(ceph_pool_objects{cluster=~\\"$cluster\\", }[1m]) *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{name=~\\"$pool_name\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Objects per second",\n "refId": "A"\n }\n ],\n "title": "$pool_name Object Ingress/Egress",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "iops"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 12,\n "x": 0,\n "y": 7\n },\n "id": 5,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "reads",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "rate(ceph_pool_rd{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{name=~\\"$pool_name\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "reads",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_pool_wr{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{name=~\\"$pool_name\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "writes",\n "refId": "B"\n }\n ],\n "title": "$pool_name Client IOPS",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "Bps"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 12,\n "x": 12,\n "y": 7\n },\n "id": 6,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "reads",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "rate(ceph_pool_rd_bytes{cluster=~\\"$cluster\\", }[$__rate_interval]) +\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{name=~\\"$pool_name\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "reads",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_pool_wr_bytes{cluster=~\\"$cluster\\", }[$__rate_interval]) +\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{name=~\\"$pool_name\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "writes",\n "refId": "B"\n }\n ],\n "title": "$pool_name Client Throughput",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Objects",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 12,\n "x": 0,\n "y": 14\n },\n "id": 7,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "ceph_pool_objects{cluster=~\\"$cluster\\", } *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{name=~\\"$pool_name\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Number of Objects",\n "refId": "A"\n }\n ],\n "title": "$pool_name Objects",\n "type": "timeseries"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 22,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "Pool Name",\n "multi": false,\n "name": "pool_name",\n "options": [ ],\n "query": "label_values(ceph_pool_metadata{cluster=~\\"$cluster\\", }, name)",\n "refresh": 1,\n "regex": "",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "Ceph Pool Details",\n "uid": "-xyV8KCiz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/ceph-cluster.json': '{\n "__requires": [\n {\n "type": "grafana",\n "id": "grafana",\n "name": "Grafana",\n "version": "5.0.0"\n },\n {\n "type": "panel",\n "id": "graph",\n "name": "Graph",\n "version": "5.0.0"\n },\n {\n "type": "panel",\n "id": "heatmap",\n "name": "Heatmap",\n "version": "5.0.0"\n },\n {\n "type": "panel",\n "id": "singlestat",\n "name": "Singlestat",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": []\n },\n "description": "Ceph cluster overview",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "id": null,\n "iteration": 1525415495309,\n "links": [],\n "panels": [\n {\n "cacheTimeout": null,\n "colorBackground": true,\n "colorValue": false,\n "colors": [\n "rgba(50, 128, 45, 0.9)",\n "rgba(237, 129, 40, 0.9)",\n "rgb(255, 0, 0)"\n ],\n "datasource": "$datasource",\n "editable": false,\n "error": false,\n "format": "none",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 0,\n "y": 0\n },\n "hideTimeOverride": true,\n "id": 21,\n "interval": "1m",\n "links": [],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "span": 2,\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "ceph_health_status",\n "format": "time_series",\n "instant": true,\n "interval": "$interval",\n "intervalFactor": 1,\n "refId": "A",\n "step": 60\n }\n ],\n "thresholds": "1,2",\n "timeFrom": null,\n "title": "Health Status",\n "transparent": false,\n "type": "singlestat",\n "valueFontSize": "50%",\n "valueMaps": [\n {\n "op": "=",\n "text": "OK",\n "value": "0"\n },\n {\n "op": "=",\n "text": "WARN",\n "value": "1"\n },\n {\n "op": "=",\n "text": "ERR",\n "value": "2"\n }\n ],\n "valueName": "current"\n },\n {\n "colorMode": "Panel",\n "colors": {\n "crit": "rgb(255, 0, 0)",\n "disable": "rgba(128, 128, 128, 0.9)",\n "ok": "rgba(50, 128, 45, 0.9)",\n "warn": "rgba(237, 129, 40, 0.9)"\n },\n "cornerRadius": 0,\n "datasource": "$datasource",\n "displayName": "",\n "flipCard": false,\n "flipTime": 5,\n "fontFormat": "Regular",\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 6,\n "y": 0\n },\n "id": 43,\n "isAutoScrollOnOverflow": false,\n "isGrayOnNoData": false,\n "isHideAlertsOnDisable": false,\n "isIgnoreOKColors": false,\n "links": [],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n }\n },\n "targets": [\n {\n "aggregation": "Last",\n "alias": "All",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ceph_osd_metadata)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "All",\n "refId": "A",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "In",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_osd_in)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "In",\n "refId": "B",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "Out",\n "decimals": 2,\n "displayAliasType": "Warning / Critical",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_osd_in == bool 0)",\n "format": "time_series",\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Out",\n "refId": "C",\n "units": "none",\n "valueHandler": "Number Threshold",\n "warn": 1\n },\n {\n "aggregation": "Last",\n "alias": "Up",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_osd_up)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Up",\n "refId": "D",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "Down",\n "crit": 2,\n "decimals": 2,\n "displayAliasType": "Warning / Critical",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_osd_up == bool 0)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Down",\n "refId": "E",\n "units": "none",\n "valueHandler": "Number Threshold",\n "warn": 1\n }\n ],\n "title": "OSDs",\n "type": "stat",\n "fieldConfig": {\n "defaults": {\n "mappings": [],\n "thresholds": {\n "mode": "percentage",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "All"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Out"\n },\n "properties": [\n {\n "id": "thresholds",\n "value": {\n "mode": "percentage",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "orange",\n "value": 0.1\n },\n {\n "value": 10,\n "color": "red"\n }\n ]\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Down"\n },\n "properties": [\n {\n "id": "thresholds",\n "value": {\n "mode": "percentage",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "orange",\n "value": 0.1\n },\n {\n "value": 10,\n "color": "red"\n }\n ]\n }\n }\n ]\n }\n ]\n }\n },\n {\n "clusterName": "",\n "colorMode": "Panel",\n "colors": {\n "crit": "rgba(245, 54, 54, 0.9)",\n "disable": "rgba(128, 128, 128, 0.9)",\n "ok": "rgba(50, 128, 45, 0.9)",\n "warn": "rgba(237, 129, 40, 0.9)"\n },\n "cornerRadius": 1,\n "datasource": "$datasource",\n "displayName": "",\n "flipCard": false,\n "flipTime": 5,\n "fontFormat": "Regular",\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 12,\n "y": 0\n },\n "id": 41,\n "isAutoScrollOnOverflow": false,\n "isGrayOnNoData": false,\n "isHideAlertsOnDisable": false,\n "isIgnoreOKColors": false,\n "links": [],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n }\n },\n "targets": [\n {\n "aggregation": "Last",\n "alias": "In Quorum",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_mon_quorum_status)",\n "format": "time_series",\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "In Quorum",\n "refId": "A",\n "units": "none",\n "valueHandler": "Text Only"\n },\n {\n "aggregation": "Last",\n "alias": "Total",\n "crit": 1,\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ceph_mon_quorum_status)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Total",\n "refId": "B",\n "units": "none",\n "valueHandler": "Text Only",\n "warn": 2\n },\n {\n "aggregation": "Last",\n "alias": "MONs out of Quorum",\n "crit": 1.6,\n "decimals": 2,\n "displayAliasType": "Warning / Critical",\n "displayType": "Annotation",\n "displayValueWithAlias": "Never",\n "expr": "count(ceph_mon_quorum_status) - sum(ceph_mon_quorum_status)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "MONs out of Quorum",\n "refId": "C",\n "units": "none",\n "valueHandler": "Number Threshold",\n "warn": 1.1\n }\n ],\n "title": "Monitors",\n "type": "stat"\n },\n {\n "colorMode": "Panel",\n "colors": {\n "crit": "rgba(245, 54, 54, 0.9)",\n "disable": "rgba(128, 128, 128, 0.9)",\n "ok": "rgba(50, 128, 45, 0.9)",\n "warn": "rgba(237, 129, 40, 0.9)"\n },\n "cornerRadius": 1,\n "datasource": "$datasource",\n "displayName": "",\n "flipCard": false,\n "flipTime": 5,\n "fontFormat": "Regular",\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 18,\n "y": 0\n },\n "id": 68,\n "isAutoScrollOnOverflow": false,\n "isGrayOnNoData": false,\n "isHideAlertsOnDisable": false,\n "isIgnoreOKColors": false,\n "links": [],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n }\n },\n "targets": [\n {\n "aggregation": "Last",\n "alias": "Active",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ceph_mgr_status == 1) or vector(0)",\n "format": "time_series",\n "intervalFactor": 1,\n "instant": true,\n "legendFormat": "Active",\n "refId": "A",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "Standby",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ceph_mgr_status == 0) or vector(0)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Standby",\n "refId": "B",\n "units": "none",\n "valueHandler": "Number Threshold"\n }\n ],\n "title": "MGRs",\n "type": "stat"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "decimals": 2,\n "format": "percentunit",\n "gauge": {\n "maxValue": 1,\n "minValue": 0,\n "show": true,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 6,\n "w": 6,\n "x": 0,\n "y": 6\n },\n "id": 47,\n "interval": null,\n "links": [],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": true\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum(ceph_osd_stat_bytes_used)/sum(ceph_osd_stat_bytes)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Used",\n "refId": "A"\n }\n ],\n "thresholds": "0.7,0.8",\n "title": "Capacity used",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "aliasColors": {},\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 0,\n "gridPos": {\n "h": 6,\n "w": 9,\n "x": 6,\n "y": 6\n },\n "id": 53,\n "legend": {\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "show": true,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [],\n "nullPointMode": "null",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "seriesOverrides": [\n {\n "alias": "Active",\n "color": "#508642",\n "fill": 1,\n "stack": "A"\n },\n {\n "alias": "Total",\n "color": "#f9e2d2"\n },\n {\n "alias": "Degraded",\n "color": "#eab839"\n },\n {\n "alias": "Undersized",\n "color": "#f9934e"\n },\n {\n "alias": "Inconsistent",\n "color": "#e24d42"\n },\n {\n "alias": "Down",\n "color": "#bf1b00"\n },\n {\n "alias": "Inactive",\n "color": "#bf1b00",\n "fill": 4,\n "linewidth": 0,\n "stack": "A"\n }\n ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(ceph_pg_total)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Total",\n "refId": "A"\n },\n {\n "expr": "sum(ceph_pg_active)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Active",\n "refId": "B"\n },\n {\n "expr": "sum(ceph_pg_total - ceph_pg_active)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Inactive",\n "refId": "G"\n },\n {\n "expr": "sum(ceph_pg_undersized)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Undersized",\n "refId": "F"\n },\n {\n "expr": "sum(ceph_pg_degraded)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Degraded",\n "refId": "C"\n },\n {\n "expr": "sum(ceph_pg_inconsistent)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Inconsistent",\n "refId": "D"\n },\n {\n "expr": "sum(ceph_pg_down)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Down",\n "refId": "E"\n }\n ],\n "thresholds": [],\n "timeFrom": null,\n "timeShift": null,\n "title": "PG States",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": []\n },\n "yaxes": [\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": "0",\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": null,\n "show": false\n }\n ]\n },\n {\n "aliasColors": {},\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "ms"\n }\n },\n "fill": 0,\n "gridPos": {\n "h": 6,\n "w": 9,\n "x": 15,\n "y": 6\n },\n "id": 66,\n "legend": {\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "show": false,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [],\n "nullPointMode": "null",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "seriesOverrides": [\n {\n "alias": "Avg Apply Latency",\n "color": "#7eb26d"\n }\n ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "quantile(0.95, ceph_osd_apply_latency_ms)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Apply Latency P_95",\n "refId": "A"\n },\n {\n "expr": "quantile(0.95, ceph_osd_commit_latency_ms)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Commit Latency P_95",\n "refId": "B"\n },\n {\n "expr": "avg(ceph_osd_apply_latency_ms)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Avg Apply Latency",\n "refId": "C"\n },\n {\n "expr": "avg(ceph_osd_commit_latency_ms)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Avg Commit Latency",\n "refId": "D"\n }\n ],\n "thresholds": [],\n "timeFrom": null,\n "timeShift": null,\n "title": "OSD Latencies",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": []\n },\n "yaxes": [\n {\n "format": "ms",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": null,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": null,\n "show": true\n }\n ]\n },\n {\n "aliasColors": {},\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "Bps"\n }\n },\n "fill": 1,\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 0,\n "y": 9\n },\n "id": 45,\n "legend": {\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "show": true,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [],\n "nullPointMode": "null",\n "percentage": false,\n "pointradius": 0.5,\n "points": false,\n "renderer": "flot",\n "seriesOverrides": [\n {\n "alias": "Reads",\n "transform": "negative-Y"\n }\n ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(irate(ceph_osd_op_w_in_bytes[1m]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Writes",\n "refId": "A"\n },\n {\n "expr": "sum(irate(ceph_osd_op_r_out_bytes[1m]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Reads",\n "refId": "B"\n }\n ],\n "thresholds": [],\n "timeFrom": null,\n "timeShift": null,\n "title": "Cluster I/O",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": []\n },\n "yaxes": [\n {\n "format": "Bps",\n "label": "Read (-) / Write (+)",\n "logBase": 1,\n "max": null,\n "min": null,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": null,\n "show": false\n }\n ]\n },\n {\n "aliasColors": {},\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "Bps"\n }\n },\n "fill": 1,\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 12,\n "y": 9\n },\n "id": 62,\n "legend": {\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "show": false,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [],\n "nullPointMode": "null",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "seriesOverrides": [],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(deriv(ceph_pool_stored[1m]))",\n "format": "time_series",\n "intervalFactor": 1,\n "refId": "A"\n }\n ],\n "thresholds": [],\n "timeFrom": null,\n "timeShift": null,\n "title": "In-/Egress",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": []\n },\n "yaxes": [\n {\n "format": "Bps",\n "label": " Egress (-) / Ingress (+)",\n "logBase": 1,\n "max": null,\n "min": null,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": null,\n "show": false\n }\n ]\n },\n {\n "cards": {\n "cardPadding": null,\n "cardRound": 1\n },\n "color": {\n "cardColor": "rgb(0, 254, 255)",\n "colorScale": "sqrt",\n "colorScheme": "interpolateBlues",\n "exponent": 0.5,\n "min": null,\n "mode": "spectrum"\n },\n "dataFormat": "timeseries",\n "datasource": "$datasource",\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 0,\n "y": 15\n },\n "heatmap": {},\n "highlightCards": true,\n "id": 55,\n "legend": {\n "show": true\n },\n "links": [],\n "span": 12,\n "targets": [\n {\n "expr": "ceph_osd_stat_bytes_used / ceph_osd_stat_bytes",\n "format": "time_series",\n "interval": "1m",\n "intervalFactor": 1,\n "legendFormat": "Util (%)",\n "refId": "A",\n "step": 60\n }\n ],\n "timeFrom": null,\n "title": "OSD Capacity Utilization",\n "tooltip": {\n "show": true,\n "showHistogram": false\n },\n "type": "heatmap",\n "xAxis": {\n "show": true\n },\n "xBucketNumber": null,\n "xBucketSize": "",\n "yAxis": {\n "decimals": 2,\n "format": "percentunit",\n "logBase": 1,\n "max": null,\n "min": null,\n "show": true,\n "splitFactor": null\n },\n "yBucketNumber": null,\n "yBucketSize": null\n },\n {\n "cards": {\n "cardPadding": null,\n "cardRound": 1\n },\n "color": {\n "cardColor": "#b4ff00",\n "colorScale": "sqrt",\n "colorScheme": "interpolateBlues",\n "exponent": 0.5,\n "mode": "spectrum"\n },\n "dataFormat": "timeseries",\n "datasource": "$datasource",\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 6,\n "y": 15\n },\n "heatmap": {},\n "highlightCards": true,\n "id": 59,\n "legend": {\n "show": true\n },\n "links": [],\n "targets": [\n {\n "expr": "ceph_osd_numpg",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "#PGs",\n "refId": "A"\n }\n ],\n "title": "PGs per OSD",\n "tooltip": {\n "show": true,\n "showHistogram": false\n },\n "type": "heatmap",\n "xAxis": {\n "show": true\n },\n "xBucketNumber": null,\n "xBucketSize": "",\n "yAxis": {\n "decimals": null,\n "format": "none",\n "logBase": 1,\n "max": null,\n "min": null,\n "show": true,\n "splitFactor": null\n },\n "yBucketNumber": null,\n "yBucketSize": null\n },\n {\n "aliasColors": {},\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "ops"\n }\n },\n "fill": 0,\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 12,\n "y": 15\n },\n "id": 64,\n "legend": {\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "show": false,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [],\n "nullPointMode": "null",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "seriesOverrides": [],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(irate(ceph_osd_recovery_ops[1m]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Op/s",\n "refId": "A"\n }\n ],\n "thresholds": [],\n "timeFrom": null,\n "timeShift": null,\n "title": "Recovery Rate",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": []\n },\n "yaxes": [\n {\n "format": "ops",\n "label": "Recovery Ops/s",\n "logBase": 1,\n "max": null,\n "min": null,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": null,\n "show": true\n }\n ]\n }\n ],\n "refresh": "30s",\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph",\n "cluster"\n ],\n "templating": {\n "list": [\n {\n "hide": 0,\n "label": null,\n "name": "datasource",\n "options": [],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "auto": true,\n "auto_count": 10,\n "auto_min": "1m",\n "current": {\n "text": "auto",\n "value": "$__auto_interval_interval"\n },\n "datasource": null,\n "hide": 0,\n "includeAll": false,\n "label": "Interval",\n "multi": false,\n "name": "interval",\n "options": [\n {\n "selected": true,\n "text": "auto",\n "value": "$__auto_interval_interval"\n },\n {\n "selected": false,\n "text": "1m",\n "value": "1m"\n },\n {\n "selected": false,\n "text": "10m",\n "value": "10m"\n },\n {\n "selected": false,\n "text": "30m",\n "value": "30m"\n },\n {\n "selected": false,\n "text": "1h",\n "value": "1h"\n },\n {\n "selected": false,\n "text": "6h",\n "value": "6h"\n },\n {\n "selected": false,\n "text": "12h",\n "value": "12h"\n },\n {\n "selected": false,\n "text": "1d",\n "value": "1d"\n },\n {\n "selected": false,\n "text": "7d",\n "value": "7d"\n },\n {\n "selected": false,\n "text": "14d",\n "value": "14d"\n },\n {\n "selected": false,\n "text": "30d",\n "value": "30d"\n }\n ],\n "query": "1m,10m,30m,1h,6h,12h,1d,7d,14d,30d",\n "refresh": 2,\n "type": "interval"\n }\n ]\n },\n "time": {\n "from": "now-6h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "Ceph - Cluster",\n "version": 13\n}\n'}}} 2026-02-14 01:34:07,413 7fd7f125fe00 DEBUG Determined image: 'quay.io/ceph/grafana:10.4.0' 2026-02-14 01:34:07,443 7fd7f125fe00 INFO Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-grafana-trial014 2026-02-14 01:34:07,443 7fd7f125fe00 INFO /usr/bin/podman: stderr Error: no such container "ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-grafana-trial014" 2026-02-14 01:34:07,458 7fd7f125fe00 INFO Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-grafana.trial014 2026-02-14 01:34:07,458 7fd7f125fe00 INFO /usr/bin/podman: stderr Error: no such container "ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-grafana.trial014" 2026-02-14 01:34:07,458 7fd7f125fe00 INFO Deploy daemon grafana.trial014 ... 2026-02-14 01:34:09,944 7fd7f125fe00 INFO Verifying port 0.0.0.0:3000 ... 2026-02-14 01:34:10,097 7fd7f125fe00 DEBUG Non-zero exit code 1 from systemctl reset-failed ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@grafana.trial014 2026-02-14 01:34:10,097 7fd7f125fe00 DEBUG systemctl: stderr Failed to reset failed state of unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@grafana.trial014.service: Unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@grafana.trial014.service not loaded. 2026-02-14 01:34:10,224 7fd7f125fe00 DEBUG systemctl: stderr Created symlink /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460.target.wants/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@grafana.trial014.service → /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@.service. 2026-02-14 01:34:10,377 7fd7f125fe00 DEBUG firewalld.service is not enabled 2026-02-14 01:34:10,377 7fd7f125fe00 DEBUG Not possible to open ports <[3000]>. firewalld.service is not available 2026-02-14 01:34:10,653 7f4f1d64be00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.io/prometheus/prometheus:v2.51.0', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:34:10,671 7f4f1d64be00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'prometheus.trial014', 'image': '', 'deploy_arguments': [], 'params': {'tcp_ports': [9095], 'allow_ptrace': True}, 'meta': {'service_name': 'prometheus', 'ports': [9095], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'files': {'prometheus.yml': '# This file is generated by cephadm.\nglobal:\n scrape_interval: 10s\n evaluation_interval: 10s\n external_labels:\n cluster: 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\nrule_files:\n - /etc/prometheus/alerting/*\n\nalerting:\n alertmanagers:\n - scheme: http\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=alertmanager\n\nscrape_configs:\n - job_name: \'ceph\'\n honor_labels: true\n relabel_configs:\n - source_labels: [__address__]\n target_label: cluster\n replacement: 29ce0a10-0945-11f1-b2be-d404e6e7d460\n - source_labels: [instance]\n target_label: instance\n replacement: \'ceph_cluster\'\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=mgr-prometheus\n\n - job_name: \'node\'\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=node-exporter\n relabel_configs:\n - source_labels: [__address__]\n target_label: cluster\n replacement: 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\n\n - job_name: \'ceph-exporter\'\n honor_labels: true\n relabel_configs:\n - source_labels: [__address__]\n target_label: cluster\n replacement: 29ce0a10-0945-11f1-b2be-d404e6e7d460\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=ceph-exporter\n\n - job_name: \'nvmeof\'\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=nvmeof\n\n - job_name: \'nfs\'\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=nfs\n\n - job_name: \'federate\'\n scrape_interval: 15s\n honor_labels: true\n metrics_path: \'/federate\'\n params:\n \'match[]\':\n - \'{job="ceph"}\'\n - \'{job="node"}\'\n - \'{job="haproxy"}\'\n - \'{job="ceph-exporter"}\'\n static_configs:\n - targets: []\n', '/etc/prometheus/alerting/ceph_alerts.yml': 'groups:\n - name: "cluster health"\n rules:\n - alert: "CephHealthError"\n annotations:\n description: "The cluster state has been HEALTH_ERROR for more than 5 minutes. Please check \'ceph health detail\' for more information."\n summary: "Ceph is in the ERROR state"\n expr: "ceph_health_status == 2"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.2.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephHealthWarning"\n annotations:\n description: "The cluster state has been HEALTH_WARN for more than 15 minutes. Please check \'ceph health detail\' for more information."\n summary: "Ceph is in the WARNING state"\n expr: "ceph_health_status == 1"\n for: "15m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "mon"\n rules:\n - alert: "CephMonDownQuorumAtRisk"\n annotations:\n description: "{{ $min := query \\"floor(count(ceph_mon_metadata) / 2) + 1\\" | first | value }}Quorum requires a majority of monitors (x {{ $min }}) to be active. Without quorum the cluster will become inoperable, affecting all services and connected clients. The following monitors are down: {{- range query \\"(ceph_mon_quorum_status == 0) + on(ceph_daemon) group_left(hostname) (ceph_mon_metadata * 0)\\" }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-down"\n summary: "Monitor quorum is at risk"\n expr: |\n (\n (ceph_health_detail{name="MON_DOWN"} == 1) * on() (\n count(ceph_mon_quorum_status == 1) == bool (floor(count(ceph_mon_metadata) / 2) + 1)\n )\n ) == 1\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.3.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephMonDown"\n annotations:\n description: |\n {{ $down := query "count(ceph_mon_quorum_status == 0)" | first | value }}{{ $s := "" }}{{ if gt $down 1.0 }}{{ $s = "s" }}{{ end }}You have {{ $down }} monitor{{ $s }} down. Quorum is still intact, but the loss of an additional monitor will make your cluster inoperable. The following monitors are down: {{- range query "(ceph_mon_quorum_status == 0) + on(ceph_daemon) group_left(hostname) (ceph_mon_metadata * 0)" }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-down"\n summary: "One or more monitors down"\n expr: |\n count(ceph_mon_quorum_status == 0) <= (count(ceph_mon_metadata) - floor(count(ceph_mon_metadata) / 2) + 1)\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephMonDiskspaceCritical"\n annotations:\n description: "The free space available to a monitor\'s store is critically low. You should increase the space available to the monitor(s). The default directory is /var/lib/ceph/mon-*/data/store.db on traditional deployments, and /var/lib/rook/mon-*/data/store.db on the mon pod\'s worker node for Rook. Look for old, rotated versions of *.log and MANIFEST*. Do NOT touch any *.sst files. Also check any other directories under /var/lib/rook and other directories on the same filesystem, often /var/log and /var/tmp are culprits. Your monitor hosts are; {{- range query \\"ceph_mon_metadata\\"}} - {{ .Labels.hostname }} {{- end }}"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-disk-crit"\n summary: "Filesystem space on at least one monitor is critically low"\n expr: "ceph_health_detail{name=\\"MON_DISK_CRIT\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.3.2"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephMonDiskspaceLow"\n annotations:\n description: "The space available to a monitor\'s store is approaching full (>70% is the default). You should increase the space available to the monitor(s). The default directory is /var/lib/ceph/mon-*/data/store.db on traditional deployments, and /var/lib/rook/mon-*/data/store.db on the mon pod\'s worker node for Rook. Look for old, rotated versions of *.log and MANIFEST*. Do NOT touch any *.sst files. Also check any other directories under /var/lib/rook and other directories on the same filesystem, often /var/log and /var/tmp are culprits. Your monitor hosts are; {{- range query \\"ceph_mon_metadata\\"}} - {{ .Labels.hostname }} {{- end }}"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-disk-low"\n summary: "Drive space on at least one monitor is approaching full"\n expr: "ceph_health_detail{name=\\"MON_DISK_LOW\\"} == 1"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephMonClockSkew"\n annotations:\n description: "Ceph monitors rely on closely synchronized time to maintain quorum and cluster consistency. This event indicates that the time on at least one mon has drifted too far from the lead mon. Review cluster status with ceph -s. This will show which monitors are affected. Check the time sync status on each monitor host with \'ceph time-sync-status\' and the state and peers of your ntpd or chrony daemon."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-clock-skew"\n summary: "Clock skew detected among monitors"\n expr: "ceph_health_detail{name=\\"MON_CLOCK_SKEW\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "osd"\n rules:\n - alert: "CephOSDDownHigh"\n annotations:\n description: "{{ $value | humanize }}% or {{ with query \\"count(ceph_osd_up == 0)\\" }}{{ . | first | value }}{{ end }} of {{ with query \\"count(ceph_osd_up)\\" }}{{ . | first | value }}{{ end }} OSDs are down (>= 10%). The following OSDs are down: {{- range query \\"(ceph_osd_up * on(ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0\\" }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}"\n summary: "More than 10% of OSDs are down"\n expr: "count(ceph_osd_up == 0) / count(ceph_osd_up) * 100 >= 10"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephOSDHostDown"\n annotations:\n description: "The following OSDs are down: {{- range query \\"(ceph_osd_up * on(ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0\\" }} - {{ .Labels.hostname }} : {{ .Labels.ceph_daemon }} {{- end }}"\n summary: "An OSD host is offline"\n expr: "ceph_health_detail{name=\\"OSD_HOST_DOWN\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.8"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDDown"\n annotations:\n description: |\n {{ $num := query "count(ceph_osd_up == 0)" | first | value }}{{ $s := "" }}{{ if gt $num 1.0 }}{{ $s = "s" }}{{ end }}{{ $num }} OSD{{ $s }} down for over 5mins. The following OSD{{ $s }} {{ if eq $s "" }}is{{ else }}are{{ end }} down: {{- range query "(ceph_osd_up * on(ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0"}} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-down"\n summary: "An OSD has been marked down"\n expr: "ceph_health_detail{name=\\"OSD_DOWN\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.2"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDNearFull"\n annotations:\n description: "One or more OSDs have reached the NEARFULL threshold. Use \'ceph health detail\' and \'ceph osd df\' to identify the problem. To resolve, add capacity to the affected OSD\'s failure domain, restore down/out OSDs, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-nearfull"\n summary: "OSD(s) running low on free space (NEARFULL)"\n expr: "ceph_health_detail{name=\\"OSD_NEARFULL\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.3"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDFull"\n annotations:\n description: "An OSD has reached the FULL threshold. Writes to pools that share the affected OSD will be blocked. Use \'ceph health detail\' and \'ceph osd df\' to identify the problem. To resolve, add capacity to the affected OSD\'s failure domain, restore down/out OSDs, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-full"\n summary: "OSD full, writes blocked"\n expr: "ceph_health_detail{name=\\"OSD_FULL\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.6"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephOSDBackfillFull"\n annotations:\n description: "An OSD has reached the BACKFILL FULL threshold. This will prevent rebalance operations from completing. Use \'ceph health detail\' and \'ceph osd df\' to identify the problem. To resolve, add capacity to the affected OSD\'s failure domain, restore down/out OSDs, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-backfillfull"\n summary: "OSD(s) too full for backfill operations"\n expr: "ceph_health_detail{name=\\"OSD_BACKFILLFULL\\"} > 0"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDTooManyRepairs"\n annotations:\n description: "Reads from an OSD have used a secondary PG to return data to the client, indicating a potential failing drive."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-too-many-repairs"\n summary: "OSD reports a high number of read errors"\n expr: "ceph_health_detail{name=\\"OSD_TOO_MANY_REPAIRS\\"} == 1"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDTimeoutsPublicNetwork"\n annotations:\n description: "OSD heartbeats on the cluster\'s \'public\' network (frontend) are running slow. Investigate the network for latency or loss issues. Use \'ceph health detail\' to show the affected OSDs."\n summary: "Network issues delaying OSD heartbeats (public network)"\n expr: "ceph_health_detail{name=\\"OSD_SLOW_PING_TIME_FRONT\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDTimeoutsClusterNetwork"\n annotations:\n description: "OSD heartbeats on the cluster\'s \'cluster\' network (backend) are slow. Investigate the network for latency issues on this subnet. Use \'ceph health detail\' to show the affected OSDs."\n summary: "Network issues delaying OSD heartbeats (cluster network)"\n expr: "ceph_health_detail{name=\\"OSD_SLOW_PING_TIME_BACK\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDInternalDiskSizeMismatch"\n annotations:\n description: "One or more OSDs have an internal inconsistency between metadata and the size of the device. This could lead to the OSD(s) crashing in future. You should redeploy the affected OSDs."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#bluestore-disk-size-mismatch"\n summary: "OSD size inconsistency error"\n expr: "ceph_health_detail{name=\\"BLUESTORE_DISK_SIZE_MISMATCH\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephDeviceFailurePredicted"\n annotations:\n description: "The device health module has determined that one or more devices will fail soon. To review device status use \'ceph device ls\'. To show a specific device use \'ceph device info \'. Mark the OSD out so that data may migrate to other OSDs. Once the OSD has drained, destroy the OSD, replace the device, and redeploy the OSD."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#id2"\n summary: "Device(s) predicted to fail soon"\n expr: "ceph_health_detail{name=\\"DEVICE_HEALTH\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephDeviceFailurePredictionTooHigh"\n annotations:\n description: "The device health module has determined that devices predicted to fail can not be remediated automatically, since too many OSDs would be removed from the cluster to ensure performance and availability. Prevent data integrity issues by adding new OSDs so that data may be relocated."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#device-health-toomany"\n summary: "Too many devices are predicted to fail, unable to resolve"\n expr: "ceph_health_detail{name=\\"DEVICE_HEALTH_TOOMANY\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.7"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephDeviceFailureRelocationIncomplete"\n annotations:\n description: "The device health module has determined that one or more devices will fail soon, but the normal process of relocating the data on the device to other OSDs in the cluster is blocked. \\nEnsure that the cluster has available free space. It may be necessary to add capacity to the cluster to allow data from the failing device to successfully migrate, or to enable the balancer."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#device-health-in-use"\n summary: "Device failure is predicted, but unable to relocate data"\n expr: "ceph_health_detail{name=\\"DEVICE_HEALTH_IN_USE\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDFlapping"\n annotations:\n description: "OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked down and back up {{ $value | humanize }} times once a minute for 5 minutes. This may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster network, or the public network if no cluster network is deployed. Check the network stats on the listed host(s)."\n documentation: "https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds"\n summary: "Network issues are causing OSDs to flap (mark each other down)"\n expr: "(rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata) * 60 > 1"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.4"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDReadErrors"\n annotations:\n description: "An OSD has encountered read errors, but the OSD has recovered by retrying the reads. This may indicate an issue with hardware or the kernel."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#bluestore-spurious-read-errors"\n summary: "Device read errors detected"\n expr: "ceph_health_detail{name=\\"BLUESTORE_SPURIOUS_READ_ERRORS\\"} == 1"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPGImbalance"\n annotations:\n description: "OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} deviates by more than 30% from average PG count."\n summary: "PGs are not balanced across OSDs"\n expr: |\n abs(\n ((ceph_osd_numpg > 0) - on (job) group_left avg(ceph_osd_numpg > 0) by (job)) /\n on (job) group_left avg(ceph_osd_numpg > 0) by (job)\n ) * on (ceph_daemon) group_left(hostname) ceph_osd_metadata > 0.30\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.5"\n severity: "warning"\n type: "ceph_default"\n - name: "mds"\n rules:\n - alert: "CephFilesystemDamaged"\n annotations:\n description: "Filesystem metadata has been corrupted. Data may be inaccessible. Analyze metrics from the MDS daemon admin socket, or escalate to support."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages#cephfs-health-messages"\n summary: "CephFS filesystem is damaged."\n expr: "ceph_health_detail{name=\\"MDS_DAMAGE\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephFilesystemOffline"\n annotations:\n description: "All MDS ranks are unavailable. The MDS daemons managing metadata are down, rendering the filesystem offline."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-all-down"\n summary: "CephFS filesystem is offline"\n expr: "ceph_health_detail{name=\\"MDS_ALL_DOWN\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.3"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephFilesystemDegraded"\n annotations:\n description: "One or more metadata daemons (MDS ranks) are failed or in a damaged state. At best the filesystem is partially available, at worst the filesystem is completely unusable."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#fs-degraded"\n summary: "CephFS filesystem is degraded"\n expr: "ceph_health_detail{name=\\"FS_DEGRADED\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.4"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephFilesystemMDSRanksLow"\n annotations:\n description: "The filesystem\'s \'max_mds\' setting defines the number of MDS ranks in the filesystem. The current number of active MDS daemons is less than this value."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-up-less-than-max"\n summary: "Ceph MDS daemon count is lower than configured"\n expr: "ceph_health_detail{name=\\"MDS_UP_LESS_THAN_MAX\\"} > 0"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephFilesystemInsufficientStandby"\n annotations:\n description: "The minimum number of standby daemons required by standby_count_wanted is less than the current number of standby daemons. Adjust the standby count or increase the number of MDS daemons."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-insufficient-standby"\n summary: "Ceph filesystem standby daemons too few"\n expr: "ceph_health_detail{name=\\"MDS_INSUFFICIENT_STANDBY\\"} > 0"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephFilesystemFailureNoStandby"\n annotations:\n description: "An MDS daemon has failed, leaving only one active rank and no available standby. Investigate the cause of the failure or add a standby MDS."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#fs-with-failed-mds"\n summary: "MDS daemon failed, no further standby available"\n expr: "ceph_health_detail{name=\\"FS_WITH_FAILED_MDS\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.5"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephFilesystemReadOnly"\n annotations:\n description: "The filesystem has switched to READ ONLY due to an unexpected error when writing to the metadata pool. Either analyze the output from the MDS daemon admin socket, or escalate to support."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages#cephfs-health-messages"\n summary: "CephFS filesystem in read only mode due to write error(s)"\n expr: "ceph_health_detail{name=\\"MDS_HEALTH_READ_ONLY\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.2"\n severity: "critical"\n type: "ceph_default"\n - name: "mgr"\n rules:\n - alert: "CephMgrModuleCrash"\n annotations:\n description: "One or more mgr modules have crashed and have yet to be acknowledged by an administrator. A crashed module may impact functionality within the cluster. Use the \'ceph crash\' command to determine which module has failed, and archive it to acknowledge the failure."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#recent-mgr-module-crash"\n summary: "A manager module has recently crashed"\n expr: "ceph_health_detail{name=\\"RECENT_MGR_MODULE_CRASH\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.6.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephMgrPrometheusModuleInactive"\n annotations:\n description: "The mgr/prometheus module at {{ $labels.instance }} is unreachable. This could mean that the module has been disabled or the mgr daemon itself is down. Without the mgr/prometheus module metrics and alerts will no longer function. Open a shell to an admin node or toolbox pod and use \'ceph -s\' to to determine whether the mgr is active. If the mgr is not active, restart it, otherwise you can determine module status with \'ceph mgr module ls\'. If it is not listed as enabled, enable it with \'ceph mgr module enable prometheus\'."\n summary: "The mgr/prometheus module is not available"\n expr: "up{job=\\"ceph\\"} == 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.6.2"\n severity: "critical"\n type: "ceph_default"\n - name: "pgs"\n rules:\n - alert: "CephPGsInactive"\n annotations:\n description: "{{ $value }} PGs have been inactive for more than 5 minutes in pool {{ $labels.name }}. Inactive placement groups are not able to serve read/write requests."\n summary: "One or more placement groups are inactive"\n expr: "ceph_pool_metadata * on(pool_id,instance) group_left() (ceph_pg_total - ceph_pg_active) > 0"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGsUnclean"\n annotations:\n description: "{{ $value }} PGs have been unclean for more than 15 minutes in pool {{ $labels.name }}. Unclean PGs have not recovered from a previous failure."\n summary: "One or more placement groups are marked unclean"\n expr: "ceph_pool_metadata * on(pool_id,instance) group_left() (ceph_pg_total - ceph_pg_clean) > 0"\n for: "15m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.2"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPGsDamaged"\n annotations:\n description: "During data consistency checks (scrub), at least one PG has been flagged as being damaged or inconsistent. Check to see which PG is affected, and attempt a manual repair if necessary. To list problematic placement groups, use \'rados list-inconsistent-pg \'. To repair PGs use the \'ceph pg repair \' command."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-damaged"\n summary: "Placement group damaged, manual intervention needed"\n expr: "ceph_health_detail{name=~\\"PG_DAMAGED|OSD_SCRUB_ERRORS\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.4"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGRecoveryAtRisk"\n annotations:\n description: "Data redundancy is at risk since one or more OSDs are at or above the \'full\' threshold. Add more capacity to the cluster, restore down/out OSDs, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-recovery-full"\n summary: "OSDs are too full for recovery"\n expr: "ceph_health_detail{name=\\"PG_RECOVERY_FULL\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.5"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGUnavailableBlockingIO"\n annotations:\n description: "Data availability is reduced, impacting the cluster\'s ability to service I/O. One or more placement groups (PGs) are in a state that blocks I/O."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-availability"\n summary: "PG is unavailable, blocking I/O"\n expr: "((ceph_health_detail{name=\\"PG_AVAILABILITY\\"} == 1) - scalar(ceph_health_detail{name=\\"OSD_DOWN\\"})) == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.3"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGBackfillAtRisk"\n annotations:\n description: "Data redundancy may be at risk due to lack of free space within the cluster. One or more OSDs have reached the \'backfillfull\' threshold. Add more capacity, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-backfill-full"\n summary: "Backfill operations are blocked due to lack of free space"\n expr: "ceph_health_detail{name=\\"PG_BACKFILL_FULL\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.6"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGNotScrubbed"\n annotations:\n description: "One or more PGs have not been scrubbed recently. Scrubs check metadata integrity, protecting against bit-rot. They check that metadata is consistent across data replicas. When PGs miss their scrub interval, it may indicate that the scrub window is too small, or PGs were not in a \'clean\' state during the scrub window. You can manually initiate a scrub with: ceph pg scrub "\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-not-scrubbed"\n summary: "Placement group(s) have not been scrubbed"\n expr: "ceph_health_detail{name=\\"PG_NOT_SCRUBBED\\"} == 1"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPGsHighPerOSD"\n annotations:\n description: "The number of placement groups per OSD is too high (exceeds the mon_max_pg_per_osd setting).\\n Check that the pg_autoscaler has not been disabled for any pools with \'ceph osd pool autoscale-status\', and that the profile selected is appropriate. You may also adjust the target_size_ratio of a pool to guide the autoscaler based on the expected relative size of the pool (\'ceph osd pool set cephfs.cephfs.meta target_size_ratio .1\') or set the pg_autoscaler mode to \'warn\' and adjust pg_num appropriately for one or more pools."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks/#too-many-pgs"\n summary: "Placement groups per OSD is too high"\n expr: "ceph_health_detail{name=\\"TOO_MANY_PGS\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPGNotDeepScrubbed"\n annotations:\n description: "One or more PGs have not been deep scrubbed recently. Deep scrubs protect against bit-rot. They compare data replicas to ensure consistency. When PGs miss their deep scrub interval, it may indicate that the window is too small or PGs were not in a \'clean\' state during the deep-scrub window."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-not-deep-scrubbed"\n summary: "Placement group(s) have not been deep scrubbed"\n expr: "ceph_health_detail{name=\\"PG_NOT_DEEP_SCRUBBED\\"} == 1"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "nodes"\n rules:\n - alert: "CephNodeRootFilesystemFull"\n annotations:\n description: "Root volume is dangerously full: {{ $value | humanize }}% free."\n summary: "Root filesystem is dangerously full"\n expr: "node_filesystem_avail_bytes{mountpoint=\\"/\\"} / node_filesystem_size_bytes{mountpoint=\\"/\\"} * 100 < 5"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.8.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephNodeNetworkPacketDrops"\n annotations:\n description: "Node {{ $labels.instance }} experiences packet drop > 0.5% or > 10 packets/s on interface {{ $labels.device }}."\n summary: "One or more NICs reports packet drops"\n expr: |\n (\n rate(node_network_receive_drop_total{device!="lo"}[1m]) +\n rate(node_network_transmit_drop_total{device!="lo"}[1m])\n ) / (\n rate(node_network_receive_packets_total{device!="lo"}[1m]) +\n rate(node_network_transmit_packets_total{device!="lo"}[1m])\n ) >= 0.0050000000000000001 and (\n rate(node_network_receive_drop_total{device!="lo"}[1m]) +\n rate(node_network_transmit_drop_total{device!="lo"}[1m])\n ) >= 10\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.8.2"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephNodeNetworkPacketErrors"\n annotations:\n description: "Node {{ $labels.instance }} experiences packet errors > 0.01% or > 10 packets/s on interface {{ $labels.device }}."\n summary: "One or more NICs reports packet errors"\n expr: |\n (\n rate(node_network_receive_errs_total{device!="lo"}[1m]) +\n rate(node_network_transmit_errs_total{device!="lo"}[1m])\n ) / (\n rate(node_network_receive_packets_total{device!="lo"}[1m]) +\n rate(node_network_transmit_packets_total{device!="lo"}[1m])\n ) >= 0.0001 or (\n rate(node_network_receive_errs_total{device!="lo"}[1m]) +\n rate(node_network_transmit_errs_total{device!="lo"}[1m])\n ) >= 10\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.8.3"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephNodeNetworkBondDegraded"\n annotations:\n description: "Bond {{ $labels.master }} is degraded on Node {{ $labels.instance }}."\n summary: "Degraded Bond on Node {{ $labels.instance }}"\n expr: |\n node_bonding_slaves - node_bonding_active != 0\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephNodeDiskspaceWarning"\n annotations:\n description: "Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will be full in less than 5 days based on the 48 hour trailing fill rate."\n summary: "Host filesystem free space is getting low"\n expr: "predict_linear(node_filesystem_free_bytes{device=~\\"/.*\\"}[2d], 3600 * 24 * 5) *on(instance) group_left(nodename) node_uname_info < 0"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.8.4"\n severity: "warning"\n type: "ceph_default"\n - alert: CephNodeInconsistentMTU\n expr: |\n node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)\n != on (cluster, device) group_left\n quantile by (cluster, device) (\n 0.5, node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)\n )\n labels:\n severity: warning\n type: ceph_default\n annotations:\n summary: "Node {{ $labels.instance }} has inconsistent MTU settings in cluster {{ $labels.cluster }}"\n description: "Network interface {{ $labels.device }} on node {{ $labels.instance }} has MTU {{ $value }} which differs from the cluster median."\n impact: |\n - May cause packet fragmentation or packet drops\n - Risk of degraded cluster communication and performance\n - Potential instability in services relying on consistent networking (e.g., Ceph, Kubernetes)\n fix: |\n - Check the MTU of interface `{{ $labels.device }}` on node `{{ $labels.instance }}`:\n ip link show {{ $labels.device }}\n\n - Find the median MTU value across the cluster by running this PromQL query in Prometheus:\n quantile by (cluster, device) (0.5, node_network_mtu_bytes * (node_network_up{device!="lo"} > 0))\n\n - Standardize MTU across all nodes to match the median (commonly 1500 or 9000):\n ip link set dev {{ $labels.device }} mtu \n\n - Make MTU setting persistent:\n - RHEL/CentOS: edit `/etc/sysconfig/network-scripts/ifcfg-`\n - Debian/Ubuntu: edit `/etc/netplan/*.yaml` and apply with `netplan apply`\n\n - Restart the affected interface or node if required.\n - name: "pools"\n rules:\n - alert: "CephPoolGrowthWarning"\n annotations:\n description: "Pool \'{{ $labels.name }}\' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours."\n summary: "Pool growth rate may soon exceed capacity"\n expr: "(predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id, instance) group_right() ceph_pool_metadata) >= 95"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.9.2"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPoolBackfillFull"\n annotations:\n description: "A pool is approaching the near full threshold, which will prevent recovery/backfill operations from completing. Consider adding more capacity."\n summary: "Free space in a pool is too low for recovery/backfill"\n expr: "ceph_health_detail{name=\\"POOL_BACKFILLFULL\\"} > 0"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPoolFull"\n annotations:\n description: "A pool has reached its MAX quota, or OSDs supporting the pool have reached the FULL threshold. Until this is resolved, writes to the pool will be blocked. Pool Breakdown (top 5) {{- range query \\"topk(5, sort_desc(ceph_pool_percent_used * on(pool_id) group_right ceph_pool_metadata))\\" }} - {{ .Labels.name }} at {{ .Value }}% {{- end }} Increase the pool\'s quota, or add capacity to the cluster first then increase the pool\'s quota (e.g. ceph osd pool set quota max_bytes )"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pool-full"\n summary: "Pool is full - writes are blocked"\n expr: "ceph_health_detail{name=\\"POOL_FULL\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.9.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPoolNearFull"\n annotations:\n description: "A pool has exceeded the warning (percent full) threshold, or OSDs supporting the pool have reached the NEARFULL threshold. Writes may continue, but you are at risk of the pool going read-only if more capacity isn\'t made available. Determine the affected pool with \'ceph df detail\', looking at QUOTA BYTES and STORED. Increase the pool\'s quota, or add capacity to the cluster first then increase the pool\'s quota (e.g. ceph osd pool set quota max_bytes ). Also ensure that the balancer is active."\n summary: "One or more Ceph pools are nearly full"\n expr: "ceph_health_detail{name=\\"POOL_NEAR_FULL\\"} > 0"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "healthchecks"\n rules:\n - alert: "CephSlowOps"\n annotations:\n description: "{{ $value }} OSD requests are taking too long to process (osd_op_complaint_time exceeded)"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#slow-ops"\n summary: "OSD operations are slow to complete"\n expr: "ceph_healthcheck_slow_ops > 0"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephDaemonSlowOps"\n annotations:\n description: "{{ $labels.ceph_daemon }} operations are taking too long to process (complaint time exceeded)"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#slow-ops"\n summary: "{{ $labels.ceph_daemon }} operations are slow to complete"\n expr: "ceph_daemon_health_metrics{type=\\"SLOW_OPS\\"} > 0"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "cephadm"\n rules:\n - alert: "CephadmUpgradeFailed"\n annotations:\n description: "The cephadm cluster upgrade process has failed. The cluster remains in an undetermined state. Please review the cephadm logs, to understand the nature of the issue"\n summary: "Ceph version upgrade has failed"\n expr: "ceph_health_detail{name=\\"UPGRADE_EXCEPTION\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.11.2"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephadmDaemonFailed"\n annotations:\n description: "A daemon managed by cephadm is no longer active. Determine, which daemon is down with \'ceph health detail\'. you may start daemons with the \'ceph orch daemon start \'"\n summary: "A ceph daemon managed by cephadm is down"\n expr: "ceph_health_detail{name=\\"CEPHADM_FAILED_DAEMON\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.11.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephadmPaused"\n annotations:\n description: "Cluster management has been paused manually. This will prevent the orchestrator from service management and reconciliation. If this is not intentional, resume cephadm operations with \'ceph orch resume\'"\n documentation: "https://docs.ceph.com/en/latest/cephadm/operations#cephadm-paused"\n summary: "Orchestration tasks via cephadm are PAUSED"\n expr: "ceph_health_detail{name=\\"CEPHADM_PAUSED\\"} > 0"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "hardware"\n rules:\n - alert: "HardwareStorageError"\n annotations:\n description: "Some storage devices are in error. Check `ceph health detail`."\n summary: "Storage devices error(s) detected"\n expr: "ceph_health_detail{name=\\"HARDWARE_STORAGE\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwareMemoryError"\n annotations:\n description: "DIMM error(s) detected. Check `ceph health detail`."\n summary: "DIMM error(s) detected"\n expr: "ceph_health_detail{name=\\"HARDWARE_MEMORY\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.2"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwareProcessorError"\n annotations:\n description: "Processor error(s) detected. Check `ceph health detail`."\n summary: "Processor error(s) detected"\n expr: "ceph_health_detail{name=\\"HARDWARE_PROCESSOR\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.3"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwareNetworkError"\n annotations:\n description: "Network error(s) detected. Check `ceph health detail`."\n summary: "Network error(s) detected"\n expr: "ceph_health_detail{name=\\"HARDWARE_NETWORK\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.4"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwarePowerError"\n annotations:\n description: "Power supply error(s) detected. Check `ceph health detail`."\n summary: "Power supply error(s) detected"\n expr: "ceph_health_detail{name=\\"HARDWARE_POWER\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.5"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwareFanError"\n annotations:\n description: "Fan error(s) detected. Check `ceph health detail`."\n summary: "Fan error(s) detected"\n expr: "ceph_health_detail{name=\\"HARDWARE_FANS\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.6"\n severity: "critical"\n type: "ceph_default"\n - name: "PrometheusServer"\n rules:\n - alert: "PrometheusJobMissing"\n annotations:\n description: "The prometheus job that scrapes from Ceph is no longer defined, this will effectively mean you\'ll have no metrics or alerts for the cluster. Please review the job definitions in the prometheus.yml file of the prometheus instance."\n summary: "The scrape job for Ceph is missing from Prometheus"\n expr: "absent(up{job=\\"ceph\\"})"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.12.1"\n severity: "critical"\n type: "ceph_default"\n - name: "rados"\n rules:\n - alert: "CephObjectMissing"\n annotations:\n description: "The latest version of a RADOS object can not be found, even though all OSDs are up. I/O requests for this object from clients will block (hang). Resolving this issue may require the object to be rolled back to a prior version manually, and manually verified."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#object-unfound"\n summary: "Object(s) marked UNFOUND"\n expr: "(ceph_health_detail{name=\\"OBJECT_UNFOUND\\"} == 1) * on() (count(ceph_osd_up == 1) == bool count(ceph_osd_metadata)) == 1"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.1"\n severity: "critical"\n type: "ceph_default"\n - name: "generic"\n rules:\n - alert: "CephDaemonCrash"\n annotations:\n description: "One or more daemons have crashed recently, and need to be acknowledged. This notification ensures that software crashes do not go unseen. To acknowledge a crash, use the \'ceph crash archive \' command."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks/#recent-crash"\n summary: "One or more Ceph daemons have crashed, and are pending acknowledgement"\n expr: "ceph_health_detail{name=\\"RECENT_CRASH\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.1.2"\n severity: "critical"\n type: "ceph_default"\n - name: "rbdmirror"\n rules:\n - alert: "CephRBDMirrorImagesPerDaemonHigh"\n annotations:\n description: "Number of image replications per daemon is not suppossed to go beyond threshold 100"\n summary: "Number of image replications are now above 100"\n expr: "sum by (ceph_daemon, namespace) (ceph_rbd_mirror_snapshot_image_snapshots) > 100"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.2"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephRBDMirrorImagesNotInSync"\n annotations:\n description: "Both local and remote RBD mirror images should be in sync."\n summary: "Some of the RBD mirror images are not in sync with the remote counter parts."\n expr: "sum by (ceph_daemon, image, namespace, pool) (topk by (ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_local_timestamp) - topk by (ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_remote_timestamp)) != 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.3"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephRBDMirrorImagesNotInSyncVeryHigh"\n annotations:\n description: "More than 10% of the images have synchronization problems"\n summary: "Number of unsynchronized images are very high."\n expr: "count by (ceph_daemon) ((topk by (ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_local_timestamp) - topk by (ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_remote_timestamp)) != 0) > (sum by (ceph_daemon) (ceph_rbd_mirror_snapshot_snapshots)*.1)"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.4"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephRBDMirrorImageTransferBandwidthHigh"\n annotations:\n description: "Detected a heavy increase in bandwidth for rbd replications (over 80%) in the last 30 min. This might not be a problem, but it is good to review the number of images being replicated simultaneously"\n summary: "The replication network usage has been increased over 80% in the last 30 minutes. Review the number of images being replicated. This alert will be cleaned automatically after 30 minutes"\n expr: "rate(ceph_rbd_mirror_journal_replay_bytes[30m]) > 0.80"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.5"\n severity: "warning"\n type: "ceph_default"\n - name: "nvmeof"\n rules:\n - alert: "NVMeoFSubsystemNamespaceLimit"\n annotations:\n description: "Subsystems have a max namespace limit defined at creation time. This alert means that no more namespaces can be added to {{ $labels.nqn }}"\n summary: "{{ $labels.nqn }} subsystem has reached its maximum number of namespaces "\n expr: "(count by(nqn) (ceph_nvmeof_subsystem_namespace_metadata)) >= ceph_nvmeof_subsystem_namespace_limit"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFTooManyGateways"\n annotations:\n description: "You may create many gateways, but 4 is the tested limit"\n summary: "Max supported gateways exceeded "\n expr: "count(ceph_nvmeof_gateway_info) > 4.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFMaxGatewayGroupSize"\n annotations:\n description: "You may create many gateways in a gateway group, but 2 is the tested limit"\n summary: "Max gateways within a gateway group ({{ $labels.group }}) exceeded "\n expr: "count by(group) (ceph_nvmeof_gateway_info) > 2.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFSingleGatewayGroup"\n annotations:\n description: "Although a single member gateway group is valid, it should only be used for test purposes"\n summary: "The gateway group {{ $labels.group }} consists of a single gateway - HA is not possible "\n expr: "count by(group) (ceph_nvmeof_gateway_info) == 1"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighGatewayCPU"\n annotations:\n description: "Typically, high CPU may indicate degraded performance. Consider increasing the number of reactor cores"\n summary: "CPU used by {{ $labels.instance }} NVMe-oF Gateway is high "\n expr: "label_replace(avg by(instance) (rate(ceph_nvmeof_reactor_seconds_total{mode=\\"busy\\"}[1m])),\\"instance\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\") > 80.00"\n for: "10m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFGatewayOpenSecurity"\n annotations:\n description: "It is good practice to ensure subsystems use host security to reduce the risk of unexpected data loss"\n summary: "Subsystem {{ $labels.nqn }} has been defined without host level security "\n expr: "ceph_nvmeof_subsystem_metadata{allow_any_host=\\"yes\\"}"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFTooManySubsystems"\n annotations:\n description: "Although you may continue to create subsystems in {{ $labels.gateway_host }}, the configuration may not be supported"\n summary: "The number of subsystems defined to the gateway exceeds supported values "\n expr: "count by(gateway_host) (label_replace(ceph_nvmeof_subsystem_metadata,\\"gateway_host\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\")) > 16.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFVersionMismatch"\n annotations:\n description: "This may indicate an issue with deployment. Check cephadm logs"\n summary: "The cluster has different NVMe-oF gateway releases active "\n expr: "count(count by(version) (ceph_nvmeof_gateway_info)) > 1"\n for: "1h"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighClientCount"\n annotations:\n description: "The supported limit for clients connecting to a subsystem is 32"\n summary: "The number of clients connected to {{ $labels.nqn }} is too high "\n expr: "ceph_nvmeof_subsystem_host_count > 32.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighHostCPU"\n annotations:\n description: "High CPU on a gateway host can lead to CPU contention and performance degradation"\n summary: "The CPU is high ({{ $value }}%) on NVMeoF Gateway host ({{ $labels.host }}) "\n expr: "100-((100*(avg by(host) (label_replace(rate(node_cpu_seconds_total{mode=\\"idle\\"}[5m]),\\"host\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\")) * on(host) group_right label_replace(ceph_nvmeof_gateway_info,\\"host\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\")))) >= 80.00"\n for: "10m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFInterfaceDown"\n annotations:\n description: "A NIC used by one or more subsystems is in a down state"\n summary: "Network interface {{ $labels.device }} is down "\n expr: "ceph_nvmeof_subsystem_listener_iface_info{operstate=\\"down\\"}"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.14.1"\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFInterfaceDuplex"\n annotations:\n description: "Until this is resolved, performance from the gateway will be degraded"\n summary: "Network interface {{ $labels.device }} is not running in full duplex mode "\n expr: "ceph_nvmeof_subsystem_listener_iface_info{duplex!=\\"full\\"}"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighReadLatency"\n annotations:\n description: "High latencies may indicate a constraint within the cluster e.g. CPU, network. Please investigate"\n summary: "The average read latency over the last 5 mins has reached 10 ms or more on {{ $labels.gateway }}"\n expr: "label_replace((avg by(instance) ((rate(ceph_nvmeof_bdev_read_seconds_total[1m]) / rate(ceph_nvmeof_bdev_reads_completed_total[1m])))),\\"gateway\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\") > 0.01"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighWriteLatency"\n annotations:\n description: "High latencies may indicate a constraint within the cluster e.g. CPU, network. Please investigate"\n summary: "The average write latency over the last 5 mins has reached 20 ms or more on {{ $labels.gateway }}"\n expr: "label_replace((avg by(instance) ((rate(ceph_nvmeof_bdev_write_seconds_total[5m]) / rate(ceph_nvmeof_bdev_writes_completed_total[5m])))),\\"gateway\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\") > 0.02"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n', '/etc/prometheus/alerting/custom_alerts.yml': ''}, 'retention_time': '15d', 'retention_size': '0', 'ip_to_bind_to': ''}} 2026-02-14 01:34:10,671 7f4f1d64be00 DEBUG Determined image: 'quay.io/prometheus/prometheus:v2.51.0' 2026-02-14 01:34:10,703 7f4f1d64be00 INFO Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-prometheus-trial014 2026-02-14 01:34:10,704 7f4f1d64be00 INFO /usr/bin/podman: stderr Error: no such container "ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-prometheus-trial014" 2026-02-14 01:34:10,718 7f4f1d64be00 INFO Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-prometheus.trial014 2026-02-14 01:34:10,718 7f4f1d64be00 INFO /usr/bin/podman: stderr Error: no such container "ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-prometheus.trial014" 2026-02-14 01:34:10,718 7f4f1d64be00 INFO Deploy daemon prometheus.trial014 ... 2026-02-14 01:34:12,164 7f4f1d64be00 INFO Verifying port 0.0.0.0:9095 ... 2026-02-14 01:34:12,307 7f4f1d64be00 DEBUG Non-zero exit code 1 from systemctl reset-failed ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@prometheus.trial014 2026-02-14 01:34:12,307 7f4f1d64be00 DEBUG systemctl: stderr Failed to reset failed state of unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@prometheus.trial014.service: Unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@prometheus.trial014.service not loaded. 2026-02-14 01:34:12,432 7f4f1d64be00 DEBUG systemctl: stderr Created symlink /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460.target.wants/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@prometheus.trial014.service → /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@.service. 2026-02-14 01:34:12,584 7f4f1d64be00 DEBUG firewalld.service is not enabled 2026-02-14 01:34:12,584 7f4f1d64be00 DEBUG Not possible to open ports <[9095]>. firewalld.service is not available 2026-02-14 01:34:16,591 7ff0b5c38e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ls'] 2026-02-14 01:34:17,418 7f14141f6e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:34:17,641 7f49584dce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'list-networks'] 2026-02-14 01:34:28,587 7f0b7c068e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'generate-minimal-conf'] 2026-02-14 01:34:28,646 7f0b7c068e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:28,646 7f0b7c068e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:28,646 7f0b7c068e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:28,646 7f0b7c068e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config generate-minimal-conf 2026-02-14 01:34:29,212 7f5b7040ee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:34:29,229 7f5b7040ee00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'mon.trial014', 'image': '', 'deploy_arguments': [], 'params': {'reconfig': True, 'allow_ptrace': True}, 'meta': {'service_name': 'mon', 'ports': [], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n[mon.trial014]\npublic network = 10.20.192.0/20\n', 'keyring': '[mon.]\n\tkey = AQDz0I9p5ZHrBRAAqsLJRevJFlCa8pfULLdJ7Q==\n\tcaps mon = "allow *"\n', 'files': {'config': '[mon.trial014]\npublic network = 10.20.192.0/20\n'}}} 2026-02-14 01:34:29,229 7f5b7040ee00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph:squid' 2026-02-14 01:34:29,243 7f5b7040ee00 INFO Reconfig daemon mon.trial014 ... 2026-02-14 01:34:29,298 7faeba1fce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'] 2026-02-14 01:34:29,334 7f5b7040ee00 DEBUG firewalld.service is not enabled 2026-02-14 01:34:29,334 7f5b7040ee00 DEBUG Not possible to enable service . firewalld.service is not available 2026-02-14 01:34:29,337 7fc723ac7e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'set', 'client', 'debug_client', '20'] 2026-02-14 01:34:29,361 7faeba1fce00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,361 7faeba1fce00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:29,361 7faeba1fce00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,362 7faeba1fce00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config set client client_mount_timeout 600 2026-02-14 01:34:29,408 7f4216adae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'] 2026-02-14 01:34:29,429 7f5c35a0de00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'] 2026-02-14 01:34:29,438 7fc723ac7e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,438 7fc723ac7e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:29,438 7fc723ac7e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,439 7fc723ac7e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config set client debug_client 20 2026-02-14 01:34:29,481 7f4216adae00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,481 7f4216adae00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:29,481 7f4216adae00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,482 7f4216adae00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config set client debug_ms 1 2026-02-14 01:34:29,483 7f2c22c37e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:34:29,496 7fc14a840e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'] 2026-02-14 01:34:29,504 7f2c22c37e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'mgr.trial014.uucrmp', 'image': '', 'deploy_arguments': [], 'params': {'tcp_ports': [8443, 9283, 8765], 'reconfig': True, 'allow_ptrace': True}, 'meta': {'service_name': 'mgr', 'ports': [8443, 9283, 8765], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[mgr.trial014.uucrmp]\nkey = AQDz0I9pYGlEFhAAOnF6ryAzuu/H9SeCFjrELQ==\n'}} 2026-02-14 01:34:29,505 7f2c22c37e00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph:squid' 2026-02-14 01:34:29,519 7f2c22c37e00 INFO Reconfig daemon mgr.trial014.uucrmp ... 2026-02-14 01:34:29,525 7f5c35a0de00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,525 7f5c35a0de00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:29,525 7f5c35a0de00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,526 7f5c35a0de00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config set client rados_mon_op_timeout 900 2026-02-14 01:34:29,537 7fb2fa7f3e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'] 2026-02-14 01:34:29,589 7fe63ae63e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'] 2026-02-14 01:34:29,642 7f2c22c37e00 DEBUG firewalld.service is not enabled 2026-02-14 01:34:29,643 7f2c22c37e00 DEBUG Not possible to enable service . firewalld.service is not available 2026-02-14 01:34:29,648 7fca19228e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'] 2026-02-14 01:34:29,656 7f366d5e0e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'] 2026-02-14 01:34:29,657 7fc14a840e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,657 7fc14a840e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:29,657 7fc14a840e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,658 7fc14a840e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config set client rados_osd_op_timeout 900 2026-02-14 01:34:29,661 7fb2fa7f3e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,661 7fb2fa7f3e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:29,661 7fb2fa7f3e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,661 7fb2fa7f3e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config set mds debug_mds 20 2026-02-14 01:34:29,664 7f2c22c37e00 DEBUG firewalld.service is not enabled 2026-02-14 01:34:29,664 7f2c22c37e00 DEBUG Not possible to open ports <[8443, 9283, 8765]>. firewalld.service is not available 2026-02-14 01:34:29,695 7fe63ae63e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,695 7fe63ae63e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:29,695 7fe63ae63e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,696 7fe63ae63e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config set mds debug_mds_balancer 20 2026-02-14 01:34:29,751 7f677066de00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'] 2026-02-14 01:34:29,798 7fb64644ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'] 2026-02-14 01:34:29,802 7fca19228e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,802 7fca19228e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:29,803 7fca19228e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,803 7fca19228e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config set mds debug_ms 2 2026-02-14 01:34:29,831 7f366d5e0e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,832 7f366d5e0e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:29,832 7f366d5e0e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,832 7f366d5e0e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config set mds mds_debug_frag True 2026-02-14 01:34:29,839 7f5a3b1d0e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'] 2026-02-14 01:34:29,856 7f7bc2641e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'] 2026-02-14 01:34:29,866 7f677066de00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,866 7f677066de00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:29,866 7f677066de00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,867 7f677066de00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config set mds mds_debug_scatterstat True 2026-02-14 01:34:29,877 7f53ff84be00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:34:29,907 7f53ff84be00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'ceph-exporter.trial014', 'image': '', 'deploy_arguments': [], 'params': {'tcp_ports': [9926], 'reconfig': True, 'allow_ptrace': True}, 'meta': {'service_name': 'ceph-exporter', 'ports': [9926], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[client.ceph-exporter.trial014]\nkey = AQAL0Y9pSQnMEBAAd1kOyDHxNKSk3p930enz8A==\n', 'prio-limit': '5', 'stats-period': '5'}} 2026-02-14 01:34:29,907 7f53ff84be00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4' 2026-02-14 01:34:29,929 7f53ff84be00 INFO Reconfig daemon ceph-exporter.trial014 ... 2026-02-14 01:34:29,959 7f23c4fe5e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'] 2026-02-14 01:34:29,977 7fc163a26e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'] 2026-02-14 01:34:29,985 7f5a3b1d0e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,985 7f5a3b1d0e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:29,985 7f5a3b1d0e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,986 7f5a3b1d0e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config set mds mds_verify_scatter True 2026-02-14 01:34:29,986 7fb64644ae00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,986 7f7bc2641e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,986 7fb64644ae00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:29,986 7f7bc2641e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:29,986 7fb64644ae00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,986 7f7bc2641e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:29,987 7fb64644ae00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config set mds mds_op_complaint_time 180 2026-02-14 01:34:29,987 7f7bc2641e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config set mds osd_op_complaint_time 180 2026-02-14 01:34:30,015 7f9efc0bbe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'set', 'mgr', 'debug_client', '20'] 2026-02-14 01:34:30,019 7efcb8413e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'set', 'mgr', 'client_mount_timeout', '30'] 2026-02-14 01:34:30,065 7f16e1a65e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'set', 'mgr', 'debug_mgr', '20'] 2026-02-14 01:34:30,093 7f633e81fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'set', 'mgr', 'mon_warn_on_pool_no_app', 'False'] 2026-02-14 01:34:30,100 7fa5b93f2e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'set', 'mgr', 'debug_ms', '1'] 2026-02-14 01:34:30,121 7f4a0e014e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'] 2026-02-14 01:34:30,149 7f9869e4be00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'] 2026-02-14 01:34:30,165 7f53ff84be00 DEBUG firewalld.service is not enabled 2026-02-14 01:34:30,165 7f53ff84be00 DEBUG Not possible to open ports <[9926]>. firewalld.service is not available 2026-02-14 01:34:30,305 7f0e5c81ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:34:30,336 7f0e5c81ae00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'crash.trial014', 'image': '', 'deploy_arguments': [], 'params': {'reconfig': True, 'allow_ptrace': True}, 'meta': {'service_name': 'crash', 'ports': [], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[client.crash.trial014]\nkey = AQAL0Y9pPH3KOhAANHuSa4hyjyB/nHdHqJVEsg==\n'}} 2026-02-14 01:34:30,336 7f0e5c81ae00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4' 2026-02-14 01:34:30,349 7f0e5c81ae00 INFO Reconfig daemon crash.trial014 ... 2026-02-14 01:34:30,411 7fc163a26e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:30,412 7fc163a26e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:30,412 7fc163a26e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:30,412 7f633e81fe00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:30,412 7f633e81fe00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:30,412 7f633e81fe00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:30,412 7fc163a26e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config set mds rados_osd_op_timeout 900 2026-02-14 01:34:30,413 7f633e81fe00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config set mgr mon_warn_on_pool_no_app False 2026-02-14 01:34:30,424 7efcb8413e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:30,424 7efcb8413e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:30,424 7efcb8413e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:30,425 7f23c4fe5e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:30,425 7f23c4fe5e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:30,425 7efcb8413e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config set mgr client_mount_timeout 30 2026-02-14 01:34:30,425 7f23c4fe5e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:30,425 7f9869e4be00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:30,425 7f9869e4be00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:30,425 7f9869e4be00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:30,425 7f23c4fe5e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config set mds rados_mon_op_timeout 900 2026-02-14 01:34:30,426 7f9869e4be00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config set mon mon_op_complaint_time 120 2026-02-14 01:34:30,427 7f4a0e014e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:30,428 7f4a0e014e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:30,428 7f4a0e014e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:30,428 7f4a0e014e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config set osd osd_op_complaint_time 180 2026-02-14 01:34:30,429 7f9efc0bbe00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:30,429 7f9efc0bbe00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:30,429 7f9efc0bbe00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:30,429 7f9efc0bbe00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config set mgr debug_client 20 2026-02-14 01:34:30,437 7fa5b93f2e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:30,437 7fa5b93f2e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:30,437 7fa5b93f2e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:30,438 7fa5b93f2e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config set mgr debug_ms 1 2026-02-14 01:34:30,489 7f16e1a65e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:30,489 7f16e1a65e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:30,489 7f16e1a65e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:30,489 7f16e1a65e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config set mgr debug_mgr 20 2026-02-14 01:34:30,592 7fa18a1dee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.io/prometheus/alertmanager:v0.25.0', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:34:30,611 7fa18a1dee00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'alertmanager.trial014', 'image': '', 'deploy_arguments': [], 'params': {'tcp_ports': [9093, 9094], 'reconfig': True, 'allow_ptrace': True}, 'meta': {'service_name': 'alertmanager', 'ports': [9093, 9094], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'files': {'alertmanager.yml': "# This file is generated by cephadm.\n# See https://prometheus.io/docs/alerting/configuration/ for documentation.\n\nglobal:\n resolve_timeout: 5m\n http_config:\n tls_config:\n insecure_skip_verify: true\n\nroute:\n receiver: 'default'\n routes:\n - group_by: ['alertname']\n group_wait: 10s\n group_interval: 10s\n repeat_interval: 1h\n receiver: 'ceph-dashboard'\n\nreceivers:\n- name: 'default'\n webhook_configs:\n- name: 'ceph-dashboard'\n webhook_configs:\n - url: 'https://trial014:8443/api/prometheus_receiver'\n - url: 'https://trial030.front.sepia.ceph.com:8443/api/prometheus_receiver'\n"}, 'peers': ['trial014:9094']}} 2026-02-14 01:34:30,611 7fa18a1dee00 DEBUG Determined image: 'quay.io/prometheus/alertmanager:v0.25.0' 2026-02-14 01:34:30,626 7fa18a1dee00 INFO Reconfig daemon alertmanager.trial014 ... 2026-02-14 01:34:30,826 7fa18a1dee00 DEBUG firewalld.service is not enabled 2026-02-14 01:34:30,826 7fa18a1dee00 DEBUG Not possible to open ports <[9093, 9094]>. firewalld.service is not available 2026-02-14 01:34:31,032 7f98a0ed7e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'dump'] 2026-02-14 01:34:31,090 7f98a0ed7e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:31,090 7f98a0ed7e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:31,090 7f98a0ed7e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:31,090 7f98a0ed7e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config dump 2026-02-14 01:34:31,313 7ff87f1ede00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.io/ceph/grafana:10.4.0', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:34:31,372 7ff87f1ede00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'grafana.trial014', 'image': '', 'deploy_arguments': [], 'params': {'tcp_ports': [3000], 'reconfig': True, 'allow_ptrace': True}, 'meta': {'service_name': 'grafana', 'ports': [3000], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'files': {'grafana.ini': "# This file is generated by cephadm.\n[users]\n default_theme = light\n[auth.anonymous]\n enabled = true\n org_name = 'Main Org.'\n org_role = 'Viewer'\n[server]\n domain = 'bootstrap.storage.lab'\n protocol = https\n cert_file = /etc/grafana/certs/cert_file\n cert_key = /etc/grafana/certs/cert_key\n http_port = 3000\n http_addr = \n[snapshots]\n external_enabled = false\n[security]\n disable_initial_admin_creation = true\n cookie_secure = true\n cookie_samesite = none\n allow_embedding = true\n[analytics]\n check_for_updates = false\n reporting_enabled = false\n[plugins]\n check_for_plugin_updates = false\n public_key_retrieval_disabled = true", 'provisioning/datasources/ceph-dashboard.yml': "# This file is generated by cephadm.\napiVersion: 1\n\ndeleteDatasources:\n - name: 'Dashboard1'\n orgId: 1\n\ndatasources:\n - name: 'Dashboard1'\n type: 'prometheus'\n access: 'proxy'\n orgId: 1\n url: 'http://trial014:9095'\n basicAuth: false\n isDefault: true\n editable: false\n\n - name: 'Loki'\n type: 'loki'\n access: 'proxy'\n url: ''\n basicAuth: false\n isDefault: false\n editable: false", 'certs/cert_file': '# generated by cephadm\n-----BEGIN CERTIFICATE-----\nMIICyDCCAbACEQDWFuGbwrFDSaO1W/2Qonk4MA0GCSqGSIb3DQEBDQUAMCIxDTAL\nBgNVBAoMBENlcGgxETAPBgNVBAMMCHRyaWFsMDE0MB4XDTI2MDIxNDAxMzQwN1oX\nDTM2MDIxMjAxMzQwN1owIjENMAsGA1UECgwEQ2VwaDERMA8GA1UEAwwIdHJpYWww\nMTQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDAZFJgUdruhEbDg2KF\nbh+5L7LQQ0IBiFrkI9MmBZ7pVetM7Yl+VyTuJeFQ6wYsQlUywcMPpLOEcafbDh9u\nnO/NRDKX/VhaCY9+deYMgQ+XoGA3a69jdllqJQKum/s5VleUOn8TmZ1rN/Rv1cCL\no0Ip0GKuG9xKJq3PoJlKDZzS/j0fJaTnQTZNGoszD9msA+7lZjZ9WBCAntAI0a0U\nxUCvAJT8YBG4KMSh2qXufWWOunvnmG8a9hDrq/IsVHowlbfgsQ+6tBMQIuo3OElg\n5j/JWlS4erC9tYdxrSSupJcR3LO9PrlUQ+cGDSe4sX3Yr+eACXQoyJUxXBj+CAUr\na0oXAgMBAAEwDQYJKoZIhvcNAQENBQADggEBAMAu3dnDu5qLUhUezg90qIXVP19g\nUeDirCGhpwZtoR2WpkLgh1Ceq8TK0nkHUkg3P/mwKTrI2Q5OD6UPyy40kyyx2t1k\nybbTzxo9Xz6F5jOpOuyoKz18M2U3fGDLxO5lbvlLCbeQez2o+AJhcFIv1M0CxGPB\nkbFj48cvz36cxtqSTmie+yl2UzKERQ++WzBhBUSWb1cntIAlbmCbVloseWGGcrmT\nBQ3qJYW3LgVCnSG5AwcajL0dLAhrhcvERBR2MCcprFQrGZhfxMBzMXoeEoIUXiOW\nZgSbAGEF9PrOAEWmAeZOLY1xqzA5P9Ww6p5I5UY8fTSorfqh4OVSGvNtCmM=\n-----END CERTIFICATE-----\n', 'certs/cert_key': '# generated by cephadm\n-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDAZFJgUdruhEbD\ng2KFbh+5L7LQQ0IBiFrkI9MmBZ7pVetM7Yl+VyTuJeFQ6wYsQlUywcMPpLOEcafb\nDh9unO/NRDKX/VhaCY9+deYMgQ+XoGA3a69jdllqJQKum/s5VleUOn8TmZ1rN/Rv\n1cCLo0Ip0GKuG9xKJq3PoJlKDZzS/j0fJaTnQTZNGoszD9msA+7lZjZ9WBCAntAI\n0a0UxUCvAJT8YBG4KMSh2qXufWWOunvnmG8a9hDrq/IsVHowlbfgsQ+6tBMQIuo3\nOElg5j/JWlS4erC9tYdxrSSupJcR3LO9PrlUQ+cGDSe4sX3Yr+eACXQoyJUxXBj+\nCAUra0oXAgMBAAECggEADotG+EXChtzJu1wCUFNGjf7/cI5fvgfrKcCqOf9GHR5D\n/yPHN3LVCt/NjGAdWQPcDE3o4L+IW/nPllgXTjW09yUxrJsYA8PMl43IH1KJCuMH\n2F4JUJE7fzJfD94g0XEn27/JqCypDC2bCp1JGgVJ1NQOXFp6KReKyjMPg4aglV6b\nG3UZp8LDBLzEzp3Qs5jhL9YCH+ETRr5tkrZFOFJsTSckMGmzCQ5QJKsz5tDrofo9\nyzMIDjkokdSUgc8UHuXDMjWkECWRpfcQoRNVeTQ+ZND0k4HXpJEk2ecqQKF4UK6f\nf75st1wEssyWYqGfUKZcKzfVzgfAHYyF2GV6h3FmSQKBgQD4pk5plfHs2PV1L0TB\nmCMeLOKngIZmy+foM/wT/gaJ/gXSaUJ1V2syZ1/igO2jZZppKMpEApb+vMPRMnRx\nsMymLHHIrwPoQkUgdtM6aGO1IoMFyPMs9gvQLYsZh5YsltulRhoxUKDobjkjujRq\nNCKKtqEP6cTlCUMYnQxqPdsdBQKBgQDGFEbEtAiB/c9ViL9dhUkIE3sahlv/dMtZ\n7tnFDFwx2PNjQ1KezUkGAktVYgxDve4D0uXE7TbYtRdOGZsbQd8KyBAsTstorfcT\nnjCZrp3Pi2cFrOcQzlKKeztNmJOhBV4XURlP26OckElaYNZMuejUxggbvXqGqnNA\nggTXUb3VawKBgQDauOgG5vhtgJfCYjcYGKzjkVY7Q8p2SG2SC5B1QaIc5o/We/MS\n5zgLmXV1+DWR4wFF+bSayLBsGj4iGkDl9UC/+B6LCrDkMM8/9GZwXX0/y8D3aMHI\nThMsJ7PRXDyUsos5wgWuVZ+FddfbfqM2vO+Zl3bsWZVyZ4hW8WcHWawRxQKBgEzA\nOpOpdsp3wEjSTeQ86wl4EvE6MpePd+QHCtaxbV/bQqOpyG/D95GeffY9oab2By3w\nIBvoFqJtuPQIM/Oh2Hzi/HFOrr6G+f/eAdLquDddOrQWRQpcB9uk2Fu+zSVNYSID\nyJ0K7CCUP/gHIcJlHvSjLy0ghEN99F6AgsOz5qy1AoGBAPa9+KxjN5k0Rbq7hSN6\noYd72KPE2hyC1rZfmOJBYn/a5a72tIahX2BNhgyXOmB65yZpYrDaxSsqxc6RFkQK\nFv9qwxpifEQ8/AwTNYkgbkLc2WXB9LT5QCCiuE7XcOyGdGfIxvdoGwke60Rkec6f\n+3ubSlyzhTdJGepyRdWYrXSG\n-----END PRIVATE KEY-----\n', 'provisioning/dashboards/default.yml': "# This file is generated by cephadm.\napiVersion: 1\n\nproviders:\n - name: 'Ceph Dashboard'\n orgId: 1\n folder: ''\n type: file\n disableDeletion: false\n updateIntervalSeconds: 3\n editable: false\n options:\n path: '/etc/grafana/provisioning/dashboards'", '/etc/grafana/provisioning/dashboards/host-details.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.3.2"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n },\n {\n "id": "singlestat",\n "name": "Singlestat",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "$ceph_hosts System Overview",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "format": "none",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 5,\n "w": 3,\n "x": 0,\n "y": 1\n },\n "id": 3,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "count(sum by (ceph_daemon) (ceph_osd_metadata{cluster=~\\"$cluster\\", hostname=~\\"$ceph_hosts\\"}))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "OSDs",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "% Utilization",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "percent"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 10,\n "w": 6,\n "x": 3,\n "y": 1\n },\n "id": 4,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "sum by (mode) (\\n rate(node_cpu{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\", mode=~\\"(irq|nice|softirq|steal|system|user|iowait)\\"}[$__rate_interval]) or\\n rate(node_cpu_seconds_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\", mode=~\\"(irq|nice|softirq|steal|system|user|iowait)\\"}[$__rate_interval])\\n) / (\\n scalar(\\n sum(rate(node_cpu{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\n rate(node_cpu_seconds_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]))\\n ) * 100\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{mode}}",\n "refId": "A"\n }\n ],\n "title": "CPU Utilization",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "RAM used",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "bytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 10,\n "w": 6,\n "x": 9,\n "y": 1\n },\n "id": 5,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "total",\n "color": "#bf1b00",\n "fill": 0,\n "linewidth": 2,\n "stack": false\n }\n ],\n "targets": [\n {\n "expr": "node_memory_MemFree{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\nnode_memory_MemFree_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Free",\n "refId": "A"\n },\n {\n "expr": "node_memory_MemTotal{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\nnode_memory_MemTotal_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "total",\n "refId": "B"\n },\n {\n "expr": "(\\n node_memory_Cached{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\n node_memory_Cached_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n) + (\\n node_memory_Buffers{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\n node_memory_Buffers_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n) + (\\n node_memory_Slab{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\n node_memory_Slab_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "buffers/cache",\n "refId": "C"\n },\n {\n "expr": "(\\n node_memory_MemTotal{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\n node_memory_MemTotal_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n) - (\\n (\\n node_memory_MemFree{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\n node_memory_MemFree_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n ) + (\\n node_memory_Cached{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\n node_memory_Cached_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n ) + (\\n node_memory_Buffers{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\n node_memory_Buffers_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n ) + (\\n node_memory_Slab{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\n node_memory_Slab_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n )\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "used",\n "refId": "D"\n }\n ],\n "title": "RAM Usage",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Send (-) / Receive (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "decbytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 10,\n "w": 6,\n "x": 15,\n "y": 1\n },\n "id": 6,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*tx/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "sum by (device) (\\n rate(node_network_receive_bytes{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\",device!=\\"lo\\"}[$__rate_interval]) or\\n rate(node_network_receive_bytes_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\",device!=\\"lo\\"}[$__rate_interval])\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}.rx",\n "refId": "A"\n },\n {\n "expr": "sum by (device) (\\n rate(node_network_transmit_bytes{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\",device!=\\"lo\\"}[$__rate_interval]) or\\n rate(node_network_transmit_bytes_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\",device!=\\"lo\\"}[$__rate_interval])\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}.tx",\n "refId": "B"\n }\n ],\n "title": "Network Load",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Send (-) / Receive (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "pps"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 5,\n "w": 3,\n "x": 21,\n "y": 1\n },\n "id": 7,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*tx/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "rate(node_network_receive_drop{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\nrate(node_network_receive_drop_total{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}.rx",\n "refId": "A"\n },\n {\n "expr": "rate(node_network_transmit_drop{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\nrate(node_network_transmit_drop_total{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}.tx",\n "refId": "B"\n }\n ],\n "title": "Network drop rate",\n "type": "timeseries"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Each OSD consists of a Journal/WAL partition and a data partition. The RAW Capacity shown is the sum of the data partitions across all OSDs on the selected OSD hosts.",\n "format": "bytes",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 5,\n "w": 3,\n "x": 0,\n "y": 6\n },\n "id": 8,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum(\\n ceph_osd_stat_bytes{cluster=~\\"$cluster\\", } and\\n on (ceph_daemon) ceph_disk_occupation{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\", cluster=~\\"$cluster\\", }\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Raw Capacity",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Send (-) / Receive (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "pps"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 5,\n "w": 3,\n "x": 21,\n "y": 6\n },\n "id": 9,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*tx/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "rate(node_network_receive_errs{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\nrate(node_network_receive_errs_total{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}.rx",\n "refId": "A"\n },\n {\n "expr": "rate(node_network_transmit_errs{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\nrate(node_network_transmit_errs_total{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}.tx",\n "refId": "B"\n }\n ],\n "title": "Network error rate",\n "type": "timeseries"\n },\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 11\n },\n "id": 10,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "OSD Disk Performance Statistics",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "ops"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 11,\n "x": 0,\n "y": 12\n },\n "id": 11,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*reads/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "label_replace(\\n (\\n rate(node_disk_writes_completed{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\n rate(node_disk_writes_completed_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) * on(instance, device) group_left(ceph_daemon) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{cluster=~\\"$cluster\\", }, \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}({{ceph_daemon}}) writes",\n "refId": "A"\n },\n {\n "expr": "label_replace(\\n (\\n rate(node_disk_reads_completed{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\n rate(node_disk_reads_completed_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) * on(instance, device) group_left(ceph_daemon) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{cluster=~\\"$cluster\\", }, \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}({{ceph_daemon}}) reads",\n "refId": "B"\n }\n ],\n "title": "$ceph_hosts Disk IOPS",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "Bps"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 11,\n "x": 12,\n "y": 12\n },\n "id": 12,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*read/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "label_replace(\\n (\\n rate(node_disk_bytes_written{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\n rate(node_disk_written_bytes_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) * on(instance, device)\\ngroup_left(ceph_daemon) label_replace(\\n label_replace(ceph_disk_occupation_human{cluster=~\\"$cluster\\", }, \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}({{ceph_daemon}}) write",\n "refId": "A"\n },\n {\n "expr": "label_replace(\\n (\\n rate(node_disk_bytes_read{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\n rate(node_disk_read_bytes_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) * on(instance, device)\\ngroup_left(ceph_daemon) label_replace(\\n label_replace(ceph_disk_occupation_human{cluster=~\\"$cluster\\", }, \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}({{ceph_daemon}}) read",\n "refId": "B"\n }\n ],\n "title": "$ceph_hosts Throughput by Disk",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "s"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 11,\n "x": 0,\n "y": 21\n },\n "id": 13,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "max by(instance, device) (\\n label_replace(\\n (\\n (rate(node_disk_write_time_seconds_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])) /\\n clamp_min(rate(node_disk_writes_completed_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]), 0.001) or\\n (rate(node_disk_read_time_seconds_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])) /\\n clamp_min(rate(node_disk_reads_completed_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]), 0.001)\\n ),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n )\\n) * on(instance, device) group_left(ceph_daemon) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"},\\n \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}({{ceph_daemon}})",\n "refId": "A"\n }\n ],\n "title": "$ceph_hosts Disk Latency",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "%Util",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "percent"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 11,\n "x": 12,\n "y": 21\n },\n "id": 14,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "label_replace(\\n (\\n (rate(node_disk_io_time_ms{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) / 10) or\\n rate(node_disk_io_time_seconds_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) * 100\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) * on(instance, device) group_left(ceph_daemon) label_replace(\\n label_replace(ceph_disk_occupation_human{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\", cluster=~\\"$cluster\\", },\\n \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}({{ceph_daemon}})",\n "refId": "A"\n }\n ],\n "title": "$ceph_hosts Disk utilization",\n "type": "timeseries"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "null",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "instance"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Instance"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Slow Ops"\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 0,\n "y": 30\n },\n "id": 15,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "10.4.0",\n "styles": "",\n "targets": [\n {\n "expr": "topk(10,\\n (sum by (instance)(ceph_daemon_health_metrics{type=\\"SLOW_OPS\\", ceph_daemon=~\\"osd.*\\", cluster=~\\"$cluster\\", }))\\n)\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Top Slow Ops per Host",\n "transformations": [\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time": true,\n "cluster": true\n },\n "includeByName": { },\n "indexByName": { },\n "renameByName": { }\n }\n }\n ],\n "type": "table"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin",\n "overview"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": null,\n "multi": false,\n "name": "ceph_hosts",\n "options": [ ],\n "query": "label_values({__name__=~\\"ceph_.+_metadata\\", cluster=~\\"$cluster\\", }, hostname)",\n "refresh": 1,\n "regex": "([^.]*).*",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "Host Details",\n "uid": "rtOg0AiWz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/radosgw-detail.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.0.0"\n },\n {\n "id": "grafana-piechart-panel",\n "name": "Pie Chart",\n "type": "panel",\n "version": "1.3.3"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "RGW Host Detail : $rgw_servers",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "s"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 0,\n "y": 1\n },\n "id": 3,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum by (instance_id) (\\n rate(ceph_rgw_op_get_obj_lat_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n rate(ceph_rgw_op_get_obj_lat_count{cluster=~\\"$cluster\\", }[$__rate_interval])\\n) * on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "GET {{ceph_daemon}}",\n "refId": "A"\n },\n {\n "expr": "sum by (instance_id) (\\n rate(ceph_rgw_op_put_obj_lat_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n rate(ceph_rgw_op_put_obj_lat_count{cluster=~\\"$cluster\\", }[$__rate_interval])\\n) * on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "PUT {{ceph_daemon}}",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "$rgw_servers GET/PUT Latencies",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "s",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "bytes"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 8,\n "w": 7,\n "x": 6,\n "y": 1\n },\n "id": 4,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "rate(ceph_rgw_op_get_obj_bytes{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "GETs {{ceph_daemon}}",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_rgw_op_put_obj_bytes{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon)\\n ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "PUTs {{ceph_daemon}}",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Bandwidth by HTTP Operation",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "bytes",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": {\n "GETs": "#7eb26d",\n "Other": "#447ebc",\n "PUTs": "#eab839",\n "Requests": "#3f2b5b",\n "Requests Failed": "#bf1b00"\n },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 8,\n "w": 7,\n "x": 13,\n "y": 1\n },\n "id": 5,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "rate(ceph_rgw_failed_req{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Requests Failed {{ceph_daemon}}",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_rgw_get{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "GETs {{ceph_daemon}}",\n "refId": "B"\n },\n {\n "expr": "rate(ceph_rgw_put{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "PUTs {{ceph_daemon}}",\n "refId": "C"\n },\n {\n "expr": "(\\n rate(ceph_rgw_req{cluster=~\\"$cluster\\", }[$__rate_interval]) -\\n (\\n rate(ceph_rgw_get{cluster=~\\"$cluster\\", }[$__rate_interval]) +\\n rate(ceph_rgw_put{cluster=~\\"$cluster\\", }[$__rate_interval])\\n )\\n) * on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Other {{ceph_daemon}}",\n "refId": "D"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "HTTP Request Breakdown",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n }\n },\n "mappings": [ ]\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "Failures"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "#bf1b00",\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "GETs"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "#7eb26d",\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Other (HEAD,POST,DELETE)"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "#447ebc",\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "PUTs"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "#eab839",\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Requests"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "#3f2b5b",\n "mode": "fixed"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 4,\n "x": 20,\n "y": 1\n },\n "id": 6,\n "options": {\n "displayLabels": [ ],\n "legend": {\n "calcs": [ ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "values": [\n "percent",\n "value"\n ]\n },\n "pieType": "pie",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "tooltip": {\n "mode": "single",\n "sort": "none"\n }\n },\n "targets": [\n {\n "expr": "rate(ceph_rgw_failed_req{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Failures {{ceph_daemon}}",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_rgw_get{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "GETs {{ceph_daemon}}",\n "refId": "B"\n },\n {\n "expr": "rate(ceph_rgw_put{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "PUTs {{ceph_daemon}}",\n "refId": "C"\n },\n {\n "expr": "(\\n rate(ceph_rgw_req{cluster=~\\"$cluster\\", }[$__rate_interval]) -\\n (\\n rate(ceph_rgw_get{cluster=~\\"$cluster\\", }[$__rate_interval]) +\\n rate(ceph_rgw_put{cluster=~\\"$cluster\\", }[$__rate_interval])\\n )\\n) * on (instance_id) group_left (ceph_daemon)\\n ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Other (DELETE,LIST) {{ceph_daemon}}",\n "refId": "D"\n }\n ],\n "title": "Workload Breakdown",\n "type": "piechart"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin",\n "overview"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": "",\n "multi": false,\n "name": "rgw_servers",\n "options": [ ],\n "query": "label_values(ceph_rgw_metadata{cluster=~\\"$cluster\\", }, ceph_daemon)",\n "refresh": 1,\n "regex": "",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "RGW Instance Detail",\n "uid": "x5ARzZtmk",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/radosgw-overview.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.0.0"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "RGW Overview - All Gateways",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "s"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 0,\n "y": 1\n },\n "id": 3,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "label_replace(\\n rate(ceph_rgw_op_get_obj_lat_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n rate(ceph_rgw_op_get_obj_lat_count{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", },\\n \\"rgw_host\\", \\"$1\\", \\"ceph_daemon\\", \\"rgw.(.*)\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "GET {{rgw_host}}",\n "refId": "A"\n },\n {\n "expr": "label_replace(\\n rate(ceph_rgw_op_put_obj_lat_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n rate(ceph_rgw_op_put_obj_lat_count{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", },\\n \\"rgw_host\\", \\"$1\\", \\"ceph_daemon\\", \\"rgw.(.*)\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "PUT {{rgw_host}}",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Average GET/PUT Latencies by RGW Instance",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "s",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "none"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 7,\n "x": 8,\n "y": 1\n },\n "id": 4,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum by (rgw_host) (\\n label_replace(\\n rate(ceph_rgw_req{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", },\\n \\"rgw_host\\", \\"$1\\", \\"ceph_daemon\\", \\"rgw.(.*)\\"\\n )\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{rgw_host}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Total Requests/sec by RGW Instance",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "none",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "Latencies are shown stacked, without a yaxis to provide a visual indication of GET latency imbalance across RGW hosts",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "s"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 6,\n "x": 15,\n "y": 1\n },\n "id": 5,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "label_replace(\\n rate(ceph_rgw_op_get_obj_lat_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n rate(ceph_rgw_op_get_obj_lat_count{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", },\\n \\"rgw_host\\", \\"$1\\", \\"ceph_daemon\\", \\"rgw.(.*)\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{rgw_host}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "GET Latencies by RGW Instance",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "s",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "Total bytes transferred in/out of all radosgw instances within the cluster",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "bytes"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 6,\n "w": 8,\n "x": 0,\n "y": 8\n },\n "id": 6,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(rate(ceph_rgw_op_get_obj_bytes{cluster=~\\"$cluster\\", }[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "GETs",\n "refId": "A"\n },\n {\n "expr": "sum(rate(ceph_rgw_op_put_obj_bytes{cluster=~\\"$cluster\\", }[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "PUTs",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Bandwidth Consumed by Type",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "bytes",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "Total bytes transferred in/out through get/put operations, by radosgw instance",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "bytes"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 6,\n "w": 7,\n "x": 8,\n "y": 8\n },\n "id": 7,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "label_replace(sum by (instance_id) (\\n rate(ceph_rgw_op_get_obj_bytes{cluster=~\\"$cluster\\", }[$__rate_interval]) +\\n rate(ceph_rgw_op_put_obj_bytes{cluster=~\\"$cluster\\", }[$__rate_interval])) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", },\\n \\"rgw_host\\", \\"$1\\", \\"ceph_daemon\\", \\"rgw.(.*)\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{rgw_host}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Bandwidth by RGW Instance",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "bytes",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "Latencies are shown stacked, without a yaxis to provide a visual indication of PUT latency imbalance across RGW hosts",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "s"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 6,\n "w": 6,\n "x": 15,\n "y": 8\n },\n "id": 8,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "label_replace(\\n rate(ceph_rgw_op_put_obj_lat_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n rate(ceph_rgw_op_put_obj_lat_count{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", },\\n \\"rgw_host\\", \\"$1\\", \\"ceph_daemon\\", \\"rgw.(.*)\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{rgw_host}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "PUT Latencies by RGW Instance",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "s",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 12,\n "w": 9,\n "x": 0,\n "y": 12\n },\n "id": 9,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "RGW Overview - HAProxy Metrics",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 12,\n "w": 5,\n "x": 0,\n "y": 12\n },\n "id": 10,\n "legend": {\n "alignAsTable": true,\n "avg": true,\n "current": true,\n "max": true,\n "min": true,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": true\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [\n [\n {\n "alias": "/.*Back.*/",\n "transform": "negative-Y"\n },\n {\n "alias": "/.*1.*/"\n },\n {\n "alias": "/.*2.*/"\n },\n {\n "alias": "/.*3.*/"\n },\n {\n "alias": "/.*4.*/"\n },\n {\n "alias": "/.*5.*/"\n },\n {\n "alias": "/.*other.*/"\n }\n ]\n ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(\\n rate(\\n haproxy_frontend_http_responses_total{code=~\\"$code\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\", proxy=~\\"frontend\\"}[$__rate_interval]\\n )\\n) by (code)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Frontend {{ code }}",\n "refId": "A"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_backend_http_responses_total{code=~\\"$code\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\", proxy=~\\"backend\\"}[$__rate_interval]\\n )\\n) by (code)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Backend {{ code }}",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Total responses by HTTP code",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 12,\n "w": 5,\n "x": 5,\n "y": 12\n },\n "id": 11,\n "legend": {\n "alignAsTable": true,\n "avg": true,\n "current": true,\n "max": true,\n "min": true,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": true\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [\n [\n {\n "alias": "/.*Response.*/",\n "transform": "negative-Y"\n },\n {\n "alias": "/.*Backend.*/",\n "transform": "negative-Y"\n }\n ]\n ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(\\n rate(\\n haproxy_frontend_http_requests_total{proxy=~\\"frontend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Requests",\n "refId": "A"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_backend_response_errors_total{proxy=~\\"backend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "Response errors",\n "refId": "B"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_frontend_request_errors_total{proxy=~\\"frontend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Requests errors",\n "refId": "C"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_backend_redispatch_warnings_total{proxy=~\\"backend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "Backend redispatch",\n "refId": "D"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_backend_retry_warnings_total{proxy=~\\"backend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "Backend retry",\n "refId": "E"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_frontend_requests_denied_total{proxy=~\\"frontend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "Request denied",\n "refId": "F"\n },\n {\n "expr": "sum(\\n haproxy_backend_current_queue{proxy=~\\"backend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "Backend Queued",\n "refId": "G"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Total requests / responses",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 12,\n "w": 5,\n "x": 10,\n "y": 12\n },\n "id": 12,\n "legend": {\n "alignAsTable": true,\n "avg": true,\n "current": true,\n "max": true,\n "min": true,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": true\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [\n [\n {\n "alias": "/.*Back.*/",\n "transform": "negative-Y"\n }\n ]\n ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(\\n rate(\\n haproxy_frontend_connections_total{proxy=~\\"frontend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Front",\n "refId": "A"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_backend_connection_attempts_total{proxy=~\\"backend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Back",\n "refId": "B"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_backend_connection_errors_total{proxy=~\\"backend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Back errors",\n "refId": "C"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Total number of connections",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 12,\n "w": 6,\n "x": 15,\n "y": 12\n },\n "id": 13,\n "legend": {\n "alignAsTable": true,\n "avg": true,\n "current": true,\n "max": true,\n "min": true,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": true\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [\n [\n {\n "alias": "/.*OUT.*/",\n "transform": "negative-Y"\n }\n ]\n ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(\\n rate(\\n haproxy_frontend_bytes_in_total{proxy=~\\"frontend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n ) * 8\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "IN Front",\n "refId": "A"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_frontend_bytes_out_total{proxy=~\\"frontend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n ) * 8\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "OUT Front",\n "refId": "B"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_backend_bytes_in_total{proxy=~\\"backend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n ) * 8\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "IN Back",\n "refId": "C"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_backend_bytes_out_total{proxy=~\\"backend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n ) * 8\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "OUT Back",\n "refId": "D"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Current total of incoming / outgoing bytes",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin",\n "overview"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": "",\n "multi": false,\n "name": "rgw_servers",\n "options": [ ],\n "query": "label_values(ceph_rgw_metadata{cluster=~\\"$cluster\\", }, ceph_daemon)",\n "refresh": 1,\n "regex": ".*",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": "HTTP Code",\n "multi": false,\n "name": "code",\n "options": [ ],\n "query": "label_values(haproxy_server_http_responses_total{job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}, code)",\n "refresh": 1,\n "regex": "",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": ".+",\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": "job haproxy",\n "multi": true,\n "name": "job_haproxy",\n "options": [ ],\n "query": "label_values(haproxy_server_status, job)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": "Ingress Service",\n "multi": false,\n "name": "ingress_service",\n "options": [ ],\n "query": "label_values(haproxy_server_status{job=~\\"$job_haproxy\\"}, instance)",\n "refresh": 1,\n "regex": "",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "RGW Overview",\n "uid": "WAkugZpiz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/rbd-details.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.3.3"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "Detailed Performance of RBD Images (IOPS/Throughput/Latency)",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "iops"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 9,\n "w": 8,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "rate(ceph_rbd_write_ops{pool=\\"$pool\\", image=\\"$image\\", cluster=~\\"$cluster\\", }[$__rate_interval])",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{pool}} Write",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_rbd_read_ops{pool=\\"$pool\\", image=\\"$image\\", cluster=~\\"$cluster\\", }[$__rate_interval])",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{pool}} Read",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "IOPS",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "iops",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "iops",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "Bps"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 9,\n "w": 8,\n "x": 8,\n "y": 0\n },\n "id": 3,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "rate(ceph_rbd_write_bytes{pool=\\"$pool\\", image=\\"$image\\", cluster=~\\"$cluster\\", }[$__rate_interval])",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{pool}} Write",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_rbd_read_bytes{pool=\\"$pool\\", image=\\"$image\\", cluster=~\\"$cluster\\", }[$__rate_interval])",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{pool}} Read",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Throughput",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "Bps",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "Bps",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "ns"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 9,\n "w": 8,\n "x": 16,\n "y": 0\n },\n "id": 4,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "rate(ceph_rbd_write_latency_sum{pool=\\"$pool\\", image=\\"$image\\", cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n rate(ceph_rbd_write_latency_count{pool=\\"$pool\\", image=\\"$image\\", cluster=~\\"$cluster\\", }[$__rate_interval])\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{pool}} Write",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_rbd_read_latency_sum{pool=\\"$pool\\", image=\\"$image\\", cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n rate(ceph_rbd_read_latency_count{pool=\\"$pool\\", image=\\"$image\\", cluster=~\\"$cluster\\", }[$__rate_interval])\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{pool}} Read",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Average Latency",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "ns",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "ns",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "",\n "multi": false,\n "name": "pool",\n "options": [ ],\n "query": "label_values(ceph_rbd_read_ops{cluster=~\\"$cluster\\", }, pool)",\n "refresh": 1,\n "regex": "",\n "sort": 0,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "",\n "multi": false,\n "name": "image",\n "options": [ ],\n "query": "label_values(ceph_rbd_read_ops{cluster=~\\"$cluster\\", pool=\\"$pool\\"}, image)",\n "refresh": 1,\n "regex": "",\n "sort": 0,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "RBD Details",\n "uid": "YhCYGcuZz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/rgw-s3-analytics.json': '{\n "__inputs": [ ],\n "__requires": [ ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "Overview",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "colors": null,\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "decbytes"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 0,\n "y": 1\n },\n "id": 3,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_put_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Total PUTs",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "decbytes"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 6,\n "y": 1\n },\n "id": 4,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum\\n(ceph_rgw_op_get_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Total GETs",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "none"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 12,\n "y": 1\n },\n "id": 5,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_put_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Total Objects",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "decbytes"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 18,\n "y": 1\n },\n "id": 6,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum\\n((sum by(instance_id)(ceph_rgw_op_put_obj_bytes) > 0) / (sum by(instance_id)(ceph_rgw_op_put_obj_ops) > 0) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Average Object Size",\n "transparent": false,\n "type": "stat"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 0,\n "y": 4\n },\n "id": 7,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_list_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "List Objects",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_list_buckets_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "List Buckets",\n "range": true,\n "refId": "B"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_put_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Put Objects",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_per_bucket_get_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Get Objects",\n "range": false,\n "refId": "D"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_del_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Delete Objects",\n "range": false,\n "refId": "E"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_del_bucket_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Delete Buckets",\n "range": false,\n "refId": "F"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_copy_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Copy Objects",\n "range": true,\n "refId": "G"\n }\n ],\n "title": "Total Operations",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "decbytes"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 8,\n "y": 4\n },\n "id": 8,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_put_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Put Objects",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_per_bucket_get_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Get Objects",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_del_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Delete Objects",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_copy_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Copy Objects",\n "range": true,\n "refId": "D"\n }\n ],\n "title": "Total Size",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "ms"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 16,\n "y": 4\n },\n "id": 9,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_list_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "List Object",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_list_buckets_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "List Bucket",\n "range": true,\n "refId": "B"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_put_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Put Object",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_get_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Get Object",\n "range": false,\n "refId": "D"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_del_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Delete Object",\n "range": false,\n "refId": "E"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_del_bucket_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Delete Bucket",\n "range": false,\n "refId": "F"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_copy_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Copy Object",\n "range": true,\n "refId": "G"\n }\n ],\n "title": "Total Latencies",\n "type": "bargauge"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "auto",\n "cellOptions": {\n "type": "color-text"\n },\n "filterable": false,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "options": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "decbytes"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 0,\n "y": 12\n },\n "id": 10,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true,\n "sortBy": [\n {\n "desc": true,\n "displayName": "PUTs"\n }\n ]\n },\n "pluginVersion": "9.4.7",\n "styles": "",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_put_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Upload Objects",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_get_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Get Objects",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_del_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Delete Objects",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_copy_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Copy Objects",\n "range": false,\n "refId": "D"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Summary Per Bucket by Bandwidth",\n "transformations": [\n {\n "id": "merge",\n "options": { }\n },\n {\n "id": "groupBy",\n "options": {\n "fields": {\n "Bucket": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #A": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #B": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #D": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #F": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "bucket": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "ceph_daemon": {\n "aggregations": [ ],\n "operation": "groupby"\n }\n }\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time 1": true,\n "Time 2": true,\n "Time 3": true,\n "Time 4": true,\n "Time 5": true,\n "Time 6": true,\n "Time 7": true,\n "__name__ 1": true,\n "__name__ 2": true,\n "__name__ 3": true,\n "__name__ 4": true,\n "__name__ 5": true,\n "__name__ 6": true,\n "__name__ 7": true,\n "ceph_daemon 1": false,\n "ceph_daemon 2": true,\n "ceph_daemon 3": true,\n "ceph_daemon 4": true,\n "instance 1": true,\n "instance 2": true,\n "instance 3": true,\n "instance 4": true,\n "instance 5": true,\n "instance 6": true,\n "instance 7": true,\n "instance_id 1": true,\n "instance_id 2": true,\n "instance_id 3": true,\n "instance_id 4": true,\n "instance_id 5": true,\n "instance_id 6": true,\n "instance_id 7": true,\n "job 1": true,\n "job 2": true,\n "job 3": true,\n "job 4": true,\n "job 5": true,\n "job 6": true,\n "job 7": true\n },\n "indexByName": {\n "Value #A": 2,\n "Value #B": 3,\n "Value #D": 4,\n "Value #F": 5,\n "bucket": 1,\n "ceph_daemon": 0\n },\n "renameByName": {\n "Bucket": "",\n "Value #A": "PUTs",\n "Value #B": "GETs",\n "Value #C": "List",\n "Value #D": "Delete",\n "Value #E": "Copy",\n "Value #F": "Copy",\n "Value #G": "",\n "bucket": "Bucket",\n "ceph_daemon": "Daemon",\n "ceph_daemon 1": "Daemon"\n }\n }\n }\n ],\n "type": "table"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "auto",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": false,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "options": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "ms"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 12,\n "y": 12\n },\n "id": 11,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true,\n "sortBy": [\n {\n "desc": true,\n "displayName": "PUTs"\n }\n ]\n },\n "pluginVersion": "9.4.7",\n "styles": "",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_bucket_list_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "List Objects",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_bucket_put_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Upload Objects",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_bucket_get_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Get Objects",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_bucket_del_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Delete Objects",\n "range": false,\n "refId": "D"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_bucket_copy_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Copy Objects",\n "range": false,\n "refId": "E"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Latency(ms) Per Bucket",\n "transformations": [\n {\n "id": "merge",\n "options": { }\n },\n {\n "id": "joinByField",\n "options": {\n "byField": "Bucket",\n "mode": "outer"\n }\n },\n {\n "id": "groupBy",\n "options": {\n "fields": {\n "Bucket": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #A": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #B": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #C": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #D": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #F": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "bucket": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "ceph_daemon": {\n "aggregations": [ ],\n "operation": "groupby"\n }\n }\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time 1": true,\n "Time 2": true,\n "Time 3": true,\n "Time 4": true,\n "Time 5": true,\n "Time 6": true,\n "Time 7": true,\n "__name__ 1": true,\n "__name__ 2": true,\n "__name__ 3": true,\n "__name__ 4": true,\n "__name__ 5": true,\n "__name__ 6": true,\n "__name__ 7": true,\n "ceph_daemon 1": true,\n "ceph_daemon 2": true,\n "ceph_daemon 3": true,\n "ceph_daemon 4": true,\n "ceph_daemon 5": true,\n "instance 1": true,\n "instance 2": true,\n "instance 3": true,\n "instance 4": true,\n "instance 5": true,\n "instance 6": true,\n "instance 7": true,\n "instance_id 1": true,\n "instance_id 2": true,\n "instance_id 3": true,\n "instance_id 4": true,\n "instance_id 5": true,\n "instance_id 6": true,\n "instance_id 7": true,\n "job 1": true,\n "job 2": true,\n "job 3": true,\n "job 4": true,\n "job 5": true,\n "job 6": true,\n "job 7": true\n },\n "indexByName": {\n "Value #A": 2,\n "Value #B": 3,\n "Value #C": 4,\n "Value #D": 5,\n "Value #F": 6,\n "bucket": 1,\n "ceph_daemon": 0\n },\n "renameByName": {\n "Bucket": "",\n "Value #A": "PUTs",\n "Value #B": "GETs",\n "Value #C": "List",\n "Value #D": "Delete",\n "Value #E": "Copy",\n "Value #F": "Copy",\n "Value #G": "",\n "bucket": "Bucket",\n "ceph_daemon": "Daemon"\n }\n }\n }\n ],\n "type": "table"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "auto",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": false,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "options": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "decbytes"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 0,\n "y": 20\n },\n "id": 12,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true,\n "sortBy": [\n {\n "desc": true,\n "displayName": "PUTs"\n }\n ]\n },\n "pluginVersion": "9.4.7",\n "styles": "",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_put_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Upload Objects",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_get_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Get Objects",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_del_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Delete Objects",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_copy_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Copy Objects",\n "range": false,\n "refId": "D"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Summary Per User By Bandwidth",\n "transformations": [\n {\n "id": "merge",\n "options": { }\n },\n {\n "id": "groupBy",\n "options": {\n "fields": {\n "User": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #A": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #B": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #D": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #F": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "ceph_daemon": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "instance": {\n "aggregations": [ ]\n },\n "user": {\n "aggregations": [ ],\n "operation": "groupby"\n }\n }\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time 1": true,\n "Time 2": true,\n "Time 3": true,\n "Time 4": true,\n "Time 5": true,\n "Time 6": true,\n "Time 7": true,\n "__name__ 1": true,\n "__name__ 2": true,\n "__name__ 3": true,\n "__name__ 4": true,\n "__name__ 5": true,\n "__name__ 6": true,\n "__name__ 7": true,\n "ceph_daemon 1": true,\n "ceph_daemon 2": true,\n "ceph_daemon 3": true,\n "ceph_daemon 4": true,\n "instance 1": true,\n "instance 2": true,\n "instance 3": true,\n "instance 4": true,\n "instance 5": true,\n "instance 6": true,\n "instance 7": true,\n "instance_id 1": true,\n "instance_id 2": true,\n "instance_id 3": true,\n "instance_id 4": true,\n "instance_id 5": true,\n "instance_id 6": true,\n "instance_id 7": true,\n "job 1": true,\n "job 2": true,\n "job 3": true,\n "job 4": true,\n "job 5": true,\n "job 6": true,\n "job 7": true\n },\n "indexByName": {\n "Value #A": 2,\n "Value #B": 3,\n "Value #D": 4,\n "Value #F": 5,\n "ceph_daemon": 0,\n "user": 1\n },\n "renameByName": {\n "Bucket": "",\n "Value #A": "PUTs",\n "Value #B": "GETs",\n "Value #C": "List",\n "Value #D": "Delete",\n "Value #E": "Copy",\n "Value #F": "Copy",\n "Value #G": "",\n "ceph_daemon": "Daemon",\n "user": "User"\n }\n }\n }\n ],\n "type": "table"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "auto",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": false,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "options": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "ms"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 12,\n "y": 20\n },\n "id": 13,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true,\n "sortBy": [\n {\n "desc": true,\n "displayName": "PUTs"\n }\n ]\n },\n "pluginVersion": "9.4.7",\n "styles": "",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_list_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_put_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_get_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_del_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "D"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_copy_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "E"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Latency(ms) Per User",\n "transformations": [\n {\n "id": "merge",\n "options": { }\n },\n {\n "id": "joinByField",\n "options": {\n "byField": "User",\n "mode": "outer"\n }\n },\n {\n "id": "groupBy",\n "options": {\n "fields": {\n "User": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #A": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #B": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #C": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #D": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #F": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "ceph_daemon": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "user": {\n "aggregations": [ ],\n "operation": "groupby"\n }\n }\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time 1": true,\n "Time 2": true,\n "Time 3": true,\n "Time 4": true,\n "Time 5": true,\n "Time 6": true,\n "Time 7": true,\n "__name__ 1": true,\n "__name__ 2": true,\n "__name__ 3": true,\n "__name__ 4": true,\n "__name__ 5": true,\n "__name__ 6": true,\n "__name__ 7": true,\n "ceph_daemon 1": true,\n "ceph_daemon 2": true,\n "ceph_daemon 3": true,\n "ceph_daemon 4": true,\n "ceph_daemon 5": true,\n "instance 1": true,\n "instance 2": true,\n "instance 3": true,\n "instance 4": true,\n "instance 5": true,\n "instance 6": true,\n "instance 7": true,\n "instance_id 1": true,\n "instance_id 2": true,\n "instance_id 3": true,\n "instance_id 4": true,\n "instance_id 5": true,\n "instance_id 6": true,\n "instance_id 7": true,\n "job 1": true,\n "job 2": true,\n "job 3": true,\n "job 4": true,\n "job 5": true,\n "job 6": true,\n "job 7": true\n },\n "indexByName": {\n "Value #A": 2,\n "Value #B": 3,\n "Value #C": 4,\n "Value #D": 5,\n "Value #F": 6,\n "ceph_daemon": 0,\n "user": 1\n },\n "renameByName": {\n "Bucket": "",\n "Value #A": "PUTs",\n "Value #B": "GETs",\n "Value #C": "List",\n "Value #D": "Delete",\n "Value #E": "Copy",\n "Value #F": "Copy",\n "Value #G": "",\n "ceph_daemon": "Daemon",\n "user": "User"\n }\n }\n }\n ],\n "type": "table"\n },\n {\n "collapse": false,\n "collapsed": true,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 28\n },\n "id": 14,\n "panels": [\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 0,\n "y": 29\n },\n "id": 15,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "topk(5, \\n sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_put_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })\\n)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Bucket PUTs by Operations",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 6,\n "y": 29\n },\n "id": 16,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "topk(5, \\n sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_get_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })\\n)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Bucket GETs by Operations",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 12,\n "y": 29\n },\n "id": 17,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [ ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "topk(5,\\n sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_put_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })\\n)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Buckets PUTs By Size",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 18,\n "y": 29\n },\n "id": 18,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [ ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "topk(5,\\n sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_get_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })\\n)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Buckets GETs By Size",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 0,\n "y": 37\n },\n "id": 19,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_put_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket PUTs by Size",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 6,\n "y": 37\n },\n "id": 20,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_get_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket GETs by Size",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 12,\n "y": 37\n },\n "id": 21,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_copy_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket Copy by Size",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 18,\n "y": 37\n },\n "id": 22,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_del_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket Delete by Size",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 0,\n "y": 45\n },\n "id": 23,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_get_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket GETs by Operations",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 6,\n "y": 45\n },\n "id": 24,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_put_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket PUTs by Operations",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 12,\n "y": 45\n },\n "id": 25,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_list_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket List by Operations",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 18,\n "y": 45\n },\n "id": 26,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_del_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket Delete by Operations",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 0,\n "y": 53\n },\n "id": 27,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_copy_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket Copy by Operations",\n "type": "timeseries"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "auto",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": false,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "options": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "none"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 12,\n "y": 53\n },\n "id": 28,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true,\n "sortBy": [\n {\n "desc": true,\n "displayName": "PUTs"\n }\n ]\n },\n "pluginVersion": "9.4.7",\n "styles": "",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_put_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_get_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_del_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_copy_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "D"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_list_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "E"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Summary Per Bucket by Operations",\n "transformations": [\n {\n "id": "merge",\n "options": { }\n },\n {\n "id": "joinByField",\n "options": {\n "byField": "Bucket",\n "mode": "outer"\n }\n },\n {\n "id": "groupBy",\n "options": {\n "fields": {\n "Bucket": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #A": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #B": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #C": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #D": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #F": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "bucket": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "ceph_daemon": {\n "aggregations": [ ],\n "operation": "groupby"\n }\n }\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time 1": true,\n "Time 2": true,\n "Time 3": true,\n "Time 4": true,\n "Time 5": true,\n "Time 6": true,\n "Time 7": true,\n "__name__": true,\n "__name__ 1": true,\n "__name__ 2": true,\n "__name__ 3": true,\n "__name__ 4": true,\n "__name__ 5": true,\n "__name__ 6": true,\n "__name__ 7": true,\n "ceph_daemon 1": true,\n "ceph_daemon 2": true,\n "ceph_daemon 3": true,\n "ceph_daemon 4": true,\n "instance 1": true,\n "instance 2": true,\n "instance 3": true,\n "instance 4": true,\n "instance 5": true,\n "instance 6": true,\n "instance 7": true,\n "instance_id 1": true,\n "instance_id 2": true,\n "instance_id 3": true,\n "instance_id 4": true,\n "instance_id 5": true,\n "instance_id 6": true,\n "instance_id 7": true,\n "job 1": true,\n "job 2": true,\n "job 3": true,\n "job 4": true,\n "job 5": true,\n "job 6": true,\n "job 7": true\n },\n "indexByName": {\n "Value #A": 2,\n "Value #B": 3,\n "Value #C": 4,\n "Value #D": 5,\n "Value #F": 6,\n "bucket": 1,\n "ceph_daemon": 0\n },\n "renameByName": {\n "Bucket": "",\n "Value #A": "PUTs",\n "Value #B": "GETs",\n "Value #C": "List",\n "Value #D": "Delete",\n "Value #E": "Copy",\n "Value #F": "Copy",\n "Value #G": "",\n "bucket": "Bucket",\n "ceph_daemon": "Daemon"\n }\n }\n }\n ],\n "type": "table"\n }\n ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "Buckets",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "collapse": false,\n "collapsed": true,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 29\n },\n "id": 29,\n "panels": [\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 0,\n "y": 62\n },\n "id": 30,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "topk(5, \\n sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_put_obj_ops ) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })\\n)\\n",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Users PUTs By Operations",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 6,\n "y": 62\n },\n "id": 31,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "topk(5, \\n sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_get_obj_ops ) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })\\n)\\n",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Users GETs by Operations",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 12,\n "y": 62\n },\n "id": 32,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [ ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "topk(5, \\n sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_put_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })\\n)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Users PUTs by Size",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 18,\n "y": 62\n },\n "id": 33,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [ ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "topk(5, \\n sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_get_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })\\n)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Users GETs By Size",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 0,\n "y": 70\n },\n "id": 34,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_put_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User PUTs by Size",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 6,\n "y": 70\n },\n "id": 35,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_get_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User GETs by Size",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 12,\n "y": 70\n },\n "id": 36,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_del_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User Delete by Size",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 18,\n "y": 70\n },\n "id": 37,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_copy_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User COPY by Size",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 0,\n "y": 78\n },\n "id": 38,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_get_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User GETs by Operations",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 6,\n "y": 78\n },\n "id": 39,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_put_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User PUTs by Operations",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 12,\n "y": 78\n },\n "id": 40,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_list_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User List by Operations",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 18,\n "y": 78\n },\n "id": 41,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_del_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User Delete by Operations",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 0,\n "y": 86\n },\n "id": 42,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_copy_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User Copy by Operations",\n "type": "timeseries"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "auto",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": false,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "options": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "none"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 12,\n "y": 86\n },\n "id": 43,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true,\n "sortBy": [\n {\n "desc": true,\n "displayName": "PUTs"\n }\n ]\n },\n "pluginVersion": "9.4.7",\n "styles": "",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (user, ceph_daemon) (ceph_rgw_op_per_user_put_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (user, ceph_daemon) (ceph_rgw_op_per_user_get_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (user, ceph_daemon) (ceph_rgw_op_per_user_del_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (user, ceph_daemon) (ceph_rgw_op_per_user_copy_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "D"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (user, ceph_daemon) (ceph_rgw_op_per_user_list_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "E"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Summary Per User By Operations",\n "transformations": [\n {\n "id": "merge",\n "options": { }\n },\n {\n "id": "joinByField",\n "options": {\n "byField": "User",\n "mode": "outer"\n }\n },\n {\n "id": "groupBy",\n "options": {\n "fields": {\n "User": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #A": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #B": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #C": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #D": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #F": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "ceph_daemon": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "user": {\n "aggregations": [ ],\n "operation": "groupby"\n }\n }\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": { },\n "indexByName": {\n "Value #A": 2,\n "Value #B": 3,\n "Value #C": 4,\n "Value #D": 5,\n "Value #F": 6,\n "ceph_daemon": 0,\n "user": 1\n },\n "renameByName": {\n "Value #A": "PUTs",\n "Value #B": "GETs",\n "Value #C": "LIST",\n "Value #D": "DELETE",\n "Value #F": "COPY",\n "ceph_daemon": "Daemon",\n "user": "User"\n }\n }\n }\n ],\n "type": "table"\n }\n ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "Users",\n "titleSize": "h6",\n "type": "row"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 22,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": null,\n "multi": false,\n "name": "rgw_servers",\n "options": [ ],\n "query": "label_values(ceph_rgw_metadata{cluster=~\\"$cluster\\", }, ceph_daemon)",\n "refresh": 2,\n "regex": "",\n "sort": 0,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "datasource": "$datasource",\n "hide": 2,\n "label": "filters",\n "name": "Filters",\n "type": "adhoc"\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "RGW S3 Analytics",\n "uid": "BnxelG7Sz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/hosts-overview.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.3.2"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n },\n {\n "id": "singlestat",\n "name": "Singlestat",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "format": "none",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 5,\n "w": 4,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "count(sum by (hostname) (ceph_osd_metadata{cluster=~\\"$cluster\\", }))",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "OSD Hosts",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Average CPU busy across all hosts (OSD, RGW, MON etc) within the cluster",\n "format": "percentunit",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 5,\n "w": 4,\n "x": 4,\n "y": 0\n },\n "id": 3,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "avg(1 - (\\n avg by(instance) (\\n rate(node_cpu_seconds_total{mode=\'idle\',instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}[$__rate_interval]) or\\n rate(node_cpu{mode=\'idle\',instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}[$__rate_interval])\\n )\\n))\\n",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "AVG CPU Busy",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Average Memory Usage across all hosts in the cluster (excludes buffer/cache usage)",\n "format": "percentunit",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 5,\n "w": 4,\n "x": 8,\n "y": 0\n },\n "id": 4,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "avg ((\\n (\\n node_memory_MemTotal{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"} or\\n node_memory_MemTotal_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}\\n ) - ((\\n node_memory_MemFree{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"} or\\n node_memory_MemFree_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}) +\\n (\\n node_memory_Cached{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"} or\\n node_memory_Cached_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}\\n ) + (\\n node_memory_Buffers{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"} or\\n node_memory_Buffers_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}\\n ) + (\\n node_memory_Slab{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"} or\\n node_memory_Slab_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}\\n )\\n )\\n) / (\\n node_memory_MemTotal{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"} or\\n node_memory_MemTotal_bytes{instance=~\\"($osd_hosts|$rgw_hosts|$mon_hosts|$mds_hosts).*\\"}\\n))\\n",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "AVG RAM Utilization",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "IOPS Load at the device as reported by the OS on all OSD hosts",\n "format": "none",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 5,\n "w": 4,\n "x": 12,\n "y": 0\n },\n "id": 5,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum ((\\n rate(node_disk_reads_completed{instance=~\\"($osd_hosts).*\\"}[$__rate_interval]) or\\n rate(node_disk_reads_completed_total{instance=~\\"($osd_hosts).*\\"}[$__rate_interval])\\n) + (\\n rate(node_disk_writes_completed{instance=~\\"($osd_hosts).*\\"}[$__rate_interval]) or\\n rate(node_disk_writes_completed_total{instance=~\\"($osd_hosts).*\\"}[$__rate_interval])\\n))\\n",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Physical IOPS",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Average Disk utilization for all OSD data devices (i.e. excludes journal/WAL)",\n "format": "percent",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 5,\n "w": 4,\n "x": 16,\n "y": 0\n },\n "id": 6,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "avg (\\n label_replace(\\n (rate(node_disk_io_time_ms[$__rate_interval]) / 10 ) or\\n (rate(node_disk_io_time_seconds_total[$__rate_interval]) * 100),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^.:]*).*\\"\\n ) * on(instance, device) group_left(ceph_daemon) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{instance=~\\"($osd_hosts).*\\", cluster=~\\"$cluster\\", },\\n \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^.:]*).*\\"\\n )\\n)\\n",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "AVG Disk Utilization",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Total send/receive network load across all hosts in the ceph cluster",\n "format": "bytes",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 5,\n "w": 4,\n "x": 20,\n "y": 0\n },\n "id": 7,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum (\\n (\\n rate(node_network_receive_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\",device!=\\"lo\\"}[$__rate_interval]) or\\n rate(node_network_receive_bytes_total{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\",device!=\\"lo\\"}[$__rate_interval])\\n ) unless on (device, instance)\\n label_replace((node_bonding_slaves > 0), \\"device\\", \\"$1\\", \\"master\\", \\"(.+)\\")\\n) +\\nsum (\\n (\\n rate(node_network_transmit_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\",device!=\\"lo\\"}[$__rate_interval]) or\\n rate(node_network_transmit_bytes_total{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\",device!=\\"lo\\"}[$__rate_interval])\\n ) unless on (device, instance)\\n label_replace((node_bonding_slaves > 0), \\"device\\", \\"$1\\", \\"master\\", \\"(.+)\\")\\n)\\n",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Network Load",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "percent"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 0,\n "y": 5\n },\n "id": 8,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "topk(10,\\n 100 * (\\n 1 - (\\n avg by(instance) (\\n rate(node_cpu_seconds_total{mode=\'idle\',instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}[$__rate_interval]) or\\n rate(node_cpu{mode=\'idle\',instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}[$__rate_interval])\\n )\\n )\\n )\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{instance}}",\n "refId": "A"\n }\n ],\n "title": "CPU Busy - Top 10 Hosts",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "Bps"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 12,\n "y": 5\n },\n "id": 9,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "topk(10, (sum by(instance) (\\n (\\n rate(node_network_receive_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\",device!=\\"lo\\"}[$__rate_interval]) or\\n rate(node_network_receive_bytes_total{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\",device!=\\"lo\\"}[$__rate_interval])\\n ) +\\n (\\n rate(node_network_transmit_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\",device!=\\"lo\\"}[$__rate_interval]) or\\n rate(node_network_transmit_bytes_total{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\",device!=\\"lo\\"}[$__rate_interval])\\n ) unless on (device, instance)\\n label_replace((node_bonding_slaves > 0), \\"device\\", \\"$1\\", \\"master\\", \\"(.+)\\"))\\n))\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{instance}}",\n "refId": "A"\n }\n ],\n "title": "Network Load - Top 10 Hosts",\n "type": "timeseries"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": null,\n "multi": false,\n "name": "osd_hosts",\n "options": [ ],\n "query": "label_values(ceph_osd_metadata{cluster=~\\"$cluster\\", }, hostname)",\n "refresh": 1,\n "regex": "([^.]*).*",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": null,\n "multi": false,\n "name": "mon_hosts",\n "options": [ ],\n "query": "label_values(ceph_mon_metadata{cluster=~\\"$cluster\\", }, hostname)",\n "refresh": 1,\n "regex": "mon.(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": null,\n "multi": false,\n "name": "mds_hosts",\n "options": [ ],\n "query": "label_values(ceph_mds_inodes{cluster=~\\"$cluster\\"}, hostname)",\n "refresh": 1,\n "regex": "mds.(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": null,\n "multi": false,\n "name": "rgw_hosts",\n "options": [ ],\n "query": "label_values(ceph_rgw_metadata{cluster=~\\"$cluster\\"}, hostname)",\n "refresh": 1,\n "regex": "rgw.(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "Host Overview",\n "uid": "y0KGL0iZz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/pool-overview.json': '{\n "__inputs": [ ],\n "__requires": [ ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "format": "none",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "count(ceph_pool_metadata{cluster=~\\"$cluster\\", })",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Pools",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "avg"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Count of the pools that have compression enabled",\n "format": "none",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 3,\n "y": 0\n },\n "id": 3,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "count(ceph_pool_metadata{compression_mode!=\\"none\\", cluster=~\\"$cluster\\", })",\n "format": "",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Pools with Compression",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Total raw capacity available to the cluster",\n "format": "bytes",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 6,\n "y": 0\n },\n "id": 4,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum(ceph_osd_stat_bytes{cluster=~\\"$cluster\\", })",\n "format": "",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Total Raw Capacity",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Total raw capacity consumed by user data and associated overheads (metadata + redundancy)",\n "format": "bytes",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 9,\n "y": 0\n },\n "id": 5,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum(ceph_pool_bytes_used{cluster=~\\"$cluster\\", })",\n "format": "",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Raw Capacity Consumed",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Total of client data stored in the cluster",\n "format": "bytes",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 12,\n "y": 0\n },\n "id": 6,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum(ceph_pool_stored{cluster=~\\"$cluster\\", })",\n "format": "",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Logical Stored ",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "A compression saving is determined as the data eligible to be compressed minus the capacity used to store the data after compression",\n "format": "bytes",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 15,\n "y": 0\n },\n "id": 7,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum(\\n ceph_pool_compress_under_bytes{cluster=~\\"$cluster\\", } -\\n ceph_pool_compress_bytes_used{cluster=~\\"$cluster\\", }\\n)\\n",\n "format": "",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Compression Savings",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Indicates how suitable the data is within the pools that are/have been enabled for compression - averaged across all pools holding compressed data",\n "format": "percent",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 18,\n "y": 0\n },\n "id": 8,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "(\\n sum(ceph_pool_compress_under_bytes{cluster=~\\"$cluster\\", } > 0) /\\n sum(ceph_pool_stored_raw{cluster=~\\"$cluster\\", } and ceph_pool_compress_under_bytes{cluster=~\\"$cluster\\", } > 0)\\n) * 100\\n",\n "format": "table",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Compression Eligibility",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "This factor describes the average ratio of data eligible to be compressed divided by the data actually stored. It does not account for data written that was ineligible for compression (too small, or compression yield too low)",\n "format": "none",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 21,\n "y": 0\n },\n "id": 9,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum(\\n ceph_pool_compress_under_bytes{cluster=~\\"$cluster\\", } > 0)\\n / sum(ceph_pool_compress_bytes_used{cluster=~\\"$cluster\\", } > 0\\n)\\n",\n "format": "",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Compression Factor",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "auto",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "Time"\n },\n "properties": [\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "instance"\n },\n "properties": [\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "job"\n },\n "properties": [\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "name"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Pool Name"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "pool_id"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Pool ID"\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #A"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Compression Factor"\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #D"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "% Used"\n },\n {\n "id": "unit",\n "value": "percentunit"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.cellOptions",\n "value": {\n "type": "color-text"\n }\n },\n {\n "id": "thresholds",\n "value": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(245, 54, 54, 0.9)",\n "value": null\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 70\n },\n {\n "color": "rgba(50, 172, 45, 0.97)",\n "value": 85\n }\n ]\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #B"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Usable Free"\n },\n {\n "id": "unit",\n "value": "bytes"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #C"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Compression Eligibility"\n },\n {\n "id": "unit",\n "value": "percent"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #E"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Compression Savings"\n },\n {\n "id": "unit",\n "value": "bytes"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #F"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Growth (5d)"\n },\n {\n "id": "unit",\n "value": "bytes"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.cellOptions",\n "value": {\n "type": "color-text"\n }\n },\n {\n "id": "thresholds",\n "value": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(245, 54, 54, 0.9)",\n "value": null\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 70\n },\n {\n "color": "rgba(50, 172, 45, 0.97)",\n "value": 85\n }\n ]\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #G"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "IOPS"\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #H"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Bandwidth"\n },\n {\n "id": "unit",\n "value": "Bps"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "__name__"\n },\n "properties": [\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "type"\n },\n "properties": [\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "compression_mode"\n },\n "properties": [\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "description"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Type"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #J"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Stored"\n },\n {\n "id": "unit",\n "value": "bytes"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #I"\n },\n "properties": [\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #K"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Compression"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 6,\n "w": 24,\n "x": 0,\n "y": 3\n },\n "id": 10,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "10.4.0",\n "styles": "",\n "targets": [\n {\n "expr": "(\\n ceph_pool_compress_under_bytes{cluster=~\\"$cluster\\", } /\\n ceph_pool_compress_bytes_used{cluster=~\\"$cluster\\", } > 0\\n) and on(pool_id) (\\n (\\n (ceph_pool_compress_under_bytes{cluster=~\\"$cluster\\", } > 0) /\\n ceph_pool_stored_raw{cluster=~\\"$cluster\\", }\\n ) * 100 > 0.5\\n)\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "A",\n "refId": "A"\n },\n {\n "expr": "ceph_pool_max_avail{cluster=~\\"$cluster\\", } *\\n on(pool_id) group_left(name) ceph_pool_metadata{cluster=~\\"$cluster\\", }\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "B",\n "refId": "B"\n },\n {\n "expr": "(\\n (ceph_pool_compress_under_bytes{cluster=~\\"$cluster\\", } > 0) /\\n ceph_pool_stored_raw{cluster=~\\"$cluster\\", }\\n) * 100\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "C",\n "refId": "C"\n },\n {\n "expr": "ceph_pool_percent_used{cluster=~\\"$cluster\\", } *\\n on(pool_id) group_left(name) ceph_pool_metadata{cluster=~\\"$cluster\\", }\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "D",\n "refId": "D"\n },\n {\n "expr": "ceph_pool_compress_under_bytes{cluster=~\\"$cluster\\", } -\\n ceph_pool_compress_bytes_used{cluster=~\\"$cluster\\", } > 0\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "E",\n "refId": "E"\n },\n {\n "expr": "delta(ceph_pool_stored{cluster=~\\"$cluster\\", }[5d])",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "F",\n "refId": "F"\n },\n {\n "expr": "rate(ceph_pool_rd{cluster=~\\"$cluster\\", }[$__rate_interval])\\n + rate(ceph_pool_wr{cluster=~\\"$cluster\\", }[$__rate_interval])\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "G",\n "refId": "G"\n },\n {\n "expr": "rate(ceph_pool_rd_bytes{cluster=~\\"$cluster\\", }[$__rate_interval]) +\\n rate(ceph_pool_wr_bytes{cluster=~\\"$cluster\\", }[$__rate_interval])\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "H",\n "refId": "H"\n },\n {\n "expr": "ceph_pool_metadata{cluster=~\\"$cluster\\", }",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "I",\n "refId": "I"\n },\n {\n "expr": "ceph_pool_stored{cluster=~\\"$cluster\\", } * on(pool_id) group_left ceph_pool_metadata{cluster=~\\"$cluster\\", }",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "J",\n "refId": "J"\n },\n {\n "expr": "ceph_pool_metadata{compression_mode!=\\"none\\", cluster=~\\"$cluster\\", }",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "K",\n "refId": "K"\n },\n {\n "expr": "",\n "format": "",\n "intervalFactor": "",\n "legendFormat": "L",\n "refId": "L"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Pool Overview",\n "transformations": [\n {\n "id": "merge",\n "options": { }\n },\n {\n "id": "seriesToRows",\n "options": { }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time": true,\n "Value #A": true,\n "Value #B": false,\n "Value #C": true,\n "Value #D": false,\n "Value #E": true,\n "Value #I": true,\n "Value #K": true,\n "__name__": true,\n "cluster": true,\n "compression_mode": true,\n "instance": true,\n "job": true,\n "pool_id": true,\n "type": true\n },\n "includeByName": { },\n "indexByName": { },\n "renameByName": { }\n }\n }\n ],\n "type": "table"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "IOPS",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 0,\n "y": 9\n },\n "id": 11,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "topk($topk,\\n round(\\n (\\n rate(ceph_pool_rd{cluster=~\\"$cluster\\", }[$__rate_interval]) +\\n rate(ceph_pool_wr{cluster=~\\"$cluster\\", }[$__rate_interval])\\n ), 1\\n ) * on(pool_id) group_left(instance,name) ceph_pool_metadata{cluster=~\\"$cluster\\", }\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "refId": "A"\n },\n {\n "expr": "topk($topk,\\n rate(ceph_pool_wr{cluster=~\\"$cluster\\", }[$__rate_interval]) +\\n on(pool_id) group_left(instance,name) ceph_pool_metadata{cluster=~\\"$cluster\\", }\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{name}} - write",\n "refId": "B"\n }\n ],\n "title": "Top $topk Client IOPS by Pool",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Throughput",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "Bps"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 12,\n "y": 9\n },\n "id": 12,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "topk($topk,\\n (\\n rate(ceph_pool_rd_bytes{cluster=~\\"$cluster\\", }[$__rate_interval]) +\\n rate(ceph_pool_wr_bytes{cluster=~\\"$cluster\\", }[$__rate_interval])\\n ) * on(pool_id) group_left(instance, name) ceph_pool_metadata{cluster=~\\"$cluster\\", }\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "refId": "A"\n }\n ],\n "title": "Top $topk Client Bandwidth by Pool",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Capacity Used",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "bytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 24,\n "x": 0,\n "y": 17\n },\n "id": 13,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "ceph_pool_bytes_used{cluster=~\\"$cluster\\", } * on(pool_id) group_right ceph_pool_metadata{cluster=~\\"$cluster\\", }",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "refId": "A"\n }\n ],\n "title": "Pool Capacity Usage (RAW)",\n "type": "timeseries"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 22,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": {\n "text": "15",\n "value": "15"\n },\n "hide": 0,\n "includeAll": false,\n "label": "TopK",\n "multi": false,\n "name": "topk",\n "options": [\n {\n "text": "15",\n "value": "15"\n }\n ],\n "query": "15",\n "refresh": 0,\n "type": "custom"\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "Ceph Pools Overview",\n "uid": "z99hzWtmk",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/radosgw-sync-overview.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.0.0"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "Bps"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": true,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum by (source_zone) (rate(ceph_data_sync_from_zone_fetch_bytes_sum{cluster=~\\"$cluster\\", }[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{source_zone}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Replication (throughput) from Source Zone",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "Bps",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 8,\n "y": 0\n },\n "id": 3,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": true,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum by (source_zone) (rate(ceph_data_sync_from_zone_fetch_bytes_count{cluster=~\\"$cluster\\", }[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{source_zone}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Replication (objects) from Source Zone",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": "Objects/s",\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "ms"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 16,\n "y": 0\n },\n "id": 4,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": true,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum by (source_zone) (rate(ceph_data_sync_from_zone_poll_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{source_zone}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Polling Request Latency from Source Zone",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "ms",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 0,\n "y": 7\n },\n "id": 5,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": true,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum by (source_zone) (rate(ceph_data_sync_from_zone_fetch_errors{cluster=~\\"$cluster\\", }[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{source_zone}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Unsuccessful Object Replications from Source Zone",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": "Count/s",\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "s"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 16,\n "x": 8,\n "y": 7\n },\n "id": 6,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "right",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "rate(ceph_rgw_sync_delta_sync_delta[$__rate_interval])",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{instance_id}} - {{shard_id}}",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Replication(Time) Delta per shard",\n "type": "timeseries"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin",\n "overview"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": null,\n "multi": false,\n "name": "rgw_servers",\n "options": [ ],\n "query": "label_values(ceph_rgw_metadata{cluster=~\\"$cluster\\", }, ceph_daemon)",\n "refresh": 1,\n "regex": "rgw.(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "RGW Sync Overview",\n "uid": "rgw-sync-overview",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/ceph-cluster-advanced.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.3.2"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n },\n {\n "id": "heatmap",\n "name": "Heatmap",\n "type": "panel",\n "version": "5.0.0"\n },\n {\n "id": "singlestat",\n "name": "Singlestat",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "Ceph cluster overview",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "CLUSTER STATE",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "0": {\n "text": "HEALTHY"\n },\n "1": {\n "text": "WARNING"\n },\n "2": {\n "text": "ERROR"\n }\n },\n "type": "value"\n },\n {\n "id": 1,\n "options": {\n "match": null,\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "#9ac48a"\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 1\n },\n {\n "color": "rgba(245, 54, 54, 0.9)",\n "value": 2\n }\n ]\n },\n "unit": "none"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 0,\n "y": 1\n },\n "id": 3,\n "interval": "1m",\n "links": [ ],\n "options": {\n "colorMode": "value",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "ceph_health_status{cluster=~\\"$cluster\\", }",\n "format": "time_series",\n "instant": true,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Ceph health status",\n "transparent": true,\n "type": "stat"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": null,\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "max": 1,\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(245, 54, 54, 0.9)"\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 0.10000000000000001\n },\n {\n "color": "rgba(50, 172, 45, 0.97)",\n "value": 0.29999999999999999\n }\n ]\n },\n "unit": "percentunit"\n }\n },\n "gridPos": {\n "h": 6,\n "w": 3,\n "x": 3,\n "y": 1\n },\n "id": 4,\n "interval": "1m",\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "showThresholdLabels": false,\n "showThresholdMarkers": true\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "(ceph_cluster_total_bytes{cluster=~\\"$cluster\\", }-ceph_cluster_total_used_bytes{cluster=~\\"$cluster\\", })/ceph_cluster_total_bytes{cluster=~\\"$cluster\\", }",\n "format": "time_series",\n "instant": true,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Available Capacity",\n "transparent": false,\n "type": "gauge"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 2,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": null,\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(50, 172, 45, 0.97)"\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 0.025000000000000001\n },\n {\n "color": "rgba(245, 54, 54, 0.9)",\n "value": 1\n }\n ]\n },\n "unit": "bytes"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 6,\n "y": 1\n },\n "id": 5,\n "interval": "1m",\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "ceph_cluster_total_bytes{cluster=~\\"$cluster\\", }",\n "format": "time_series",\n "instant": true,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Cluster Capacity",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 1,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": null,\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "Bps"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 9,\n "y": 1\n },\n "id": 6,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(irate(ceph_osd_op_w_in_bytes{cluster=~\\"$cluster\\", }[5m]))",\n "format": "time_series",\n "instant": true,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "Write Throughput",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 1,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": null,\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "#d44a3a"\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 0\n },\n {\n "color": "#9ac48a",\n "value": 0\n }\n ]\n },\n "unit": "Bps"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 12,\n "y": 1\n },\n "id": 7,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(irate(ceph_osd_op_r_out_bytes{cluster=~\\"$cluster\\", }[5m]))",\n "format": "time_series",\n "instant": true,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "Read Throughput",\n "transparent": false,\n "type": "stat"\n },\n {\n "colorMode": "Panel",\n "colors": {\n "crit": "rgb(255, 0, 0)",\n "disable": "rgba(128, 128, 128, 0.9)",\n "ok": "rgba(50, 128, 45, 0.9)",\n "warn": "rgba(237, 129, 40, 0.9)"\n },\n "cornerRadius": 0,\n "datasource": "$datasource",\n "description": "",\n "displayName": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "All"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Out"\n },\n "properties": [\n {\n "id": "thresholds",\n "value": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 1\n }\n ]\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Down"\n },\n "properties": [\n {\n "id": "thresholds",\n "value": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 1\n }\n ]\n }\n }\n ]\n }\n ]\n },\n "flipCard": false,\n "flipTime": 5,\n "fontFormat": "Regular",\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 15,\n "y": 1\n },\n "id": 8,\n "isAutoScrollOnOverflow": false,\n "isGrayOnNoData": false,\n "isHideAlertsOnDisable": false,\n "isIgnoreOKColors": false,\n "links": [ ],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "aggregation": "Last",\n "alias": "All",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ceph_osd_metadata{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "All",\n "refId": "A",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "In",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_osd_in{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "In",\n "refId": "B",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "Out",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Warning / Critical",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_osd_in{cluster=~\\"$cluster\\", } == bool 0)",\n "format": "time_series",\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Out",\n "refId": "C",\n "units": "none",\n "valueHandler": "Number Threshold",\n "warn": 1\n },\n {\n "aggregation": "Last",\n "alias": "Up",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_osd_up{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Up",\n "refId": "D",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "Down",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Warning / Critical",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_osd_up{cluster=~\\"$cluster\\", } == bool 0)",\n "format": "time_series",\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Down",\n "refId": "E",\n "units": "none",\n "valueHandler": "Number Threshold",\n "warn": 1\n }\n ],\n "title": "OSDs",\n "transparent": false,\n "type": "stat"\n },\n {\n "colorMode": "Panel",\n "colors": {\n "crit": "rgba(245, 54, 54, 0.9)",\n "disable": "rgba(128, 128, 128, 0.9)",\n "ok": "rgba(50, 128, 45, 0.9)",\n "warn": "rgba(237, 129, 40, 0.9)"\n },\n "cornerRadius": 1,\n "datasource": "$datasource",\n "description": "",\n "displayName": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "none"\n }\n },\n "flipCard": false,\n "flipTime": 5,\n "fontFormat": "Regular",\n "gridPos": {\n "h": 6,\n "w": 3,\n "x": 21,\n "y": 1\n },\n "id": 9,\n "isAutoScrollOnOverflow": false,\n "isGrayOnNoData": false,\n "isHideAlertsOnDisable": false,\n "isIgnoreOKColors": false,\n "links": [ ],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "aggregation": "Last",\n "alias": "Active",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ceph_mgr_status{cluster=~\\"$cluster\\", } == 1) or vector(0)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Active",\n "refId": "A",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "Standby",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ceph_mgr_status{cluster=~\\"$cluster\\", } == 0) or vector(0)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Standby",\n "refId": "B",\n "units": "none",\n "valueHandler": "Number Threshold"\n }\n ],\n "title": "MGRs",\n "transparent": false,\n "type": "stat"\n },\n {\n "colorMode": "Panel",\n "colors": {\n "crit": "rgba(245, 54, 54, 0.9)",\n "disable": "rgba(128, 128, 128, 0.9)",\n "ok": "rgba(50, 128, 45, 0.9)",\n "warn": "rgba(237, 129, 40, 0.9)"\n },\n "cornerRadius": 1,\n "datasource": "$datasource",\n "description": "",\n "displayName": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 1\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "Critical"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "red",\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Warning"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "#987d24",\n "mode": "fixed"\n }\n }\n ]\n }\n ]\n },\n "flipCard": false,\n "flipTime": 5,\n "fontFormat": "Regular",\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 0,\n "y": 4\n },\n "id": 10,\n "isAutoScrollOnOverflow": false,\n "isGrayOnNoData": false,\n "isHideAlertsOnDisable": false,\n "isIgnoreOKColors": false,\n "links": [ ],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "aggregation": "Last",\n "alias": "Active",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ALERTS{alertstate=\\"firing\\",alertname=~\\"^Ceph.+\\", severity=\\"critical\\", cluster=~\\"$cluster\\", }) OR vector(0)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Critical",\n "refId": "A",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "Standby",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ALERTS{alertstate=\\"firing\\",alertname=~\\"^Ceph.+\\", severity=\\"warning\\", cluster=~\\"$cluster\\", }) OR vector(0)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Warning",\n "refId": "B",\n "units": "none",\n "valueHandler": "Number Threshold"\n }\n ],\n "title": "Firing Alerts",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "displayName": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(50, 172, 45, 0.97)",\n "value": null\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 0.025000000000000001\n },\n {\n "color": "rgba(245, 54, 54, 0.9)",\n "value": 0.10000000000000001\n }\n ]\n },\n "unit": "bytes"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 6,\n "y": 4\n },\n "id": 11,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "ceph_cluster_total_used_bytes{cluster=~\\"$cluster\\", }",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "Used Capacity",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "displayName": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "ops"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 9,\n "y": 4\n },\n "id": 12,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(irate(ceph_osd_op_w{cluster=~\\"$cluster\\", }[1m]))",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "Write IOPS",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "displayName": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "#d44a3a",\n "value": null\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 0\n },\n {\n "color": "#9ac48a",\n "value": 0\n }\n ]\n },\n "unit": "ops"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 12,\n "y": 4\n },\n "id": 13,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(irate(ceph_osd_op_r{cluster=~\\"$cluster\\", }[1m]))",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "Read IOPS",\n "transparent": false,\n "type": "stat"\n },\n {\n "colorMode": "Panel",\n "colors": {\n "crit": "rgba(245, 54, 54, 0.9)",\n "disable": "rgba(128, 128, 128, 0.9)",\n "ok": "rgba(50, 128, 45, 0.9)",\n "warn": "rgba(237, 129, 40, 0.9)"\n },\n "cornerRadius": 1,\n "datasource": "$datasource",\n "description": "",\n "displayName": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "none"\n }\n },\n "flipCard": false,\n "flipTime": 5,\n "fontFormat": "Regular",\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 15,\n "y": 4\n },\n "id": 14,\n "isAutoScrollOnOverflow": false,\n "isGrayOnNoData": false,\n "isHideAlertsOnDisable": false,\n "isIgnoreOKColors": false,\n "links": [ ],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "aggregation": "Last",\n "alias": "In Quorum",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_mon_quorum_status{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "In Quorum",\n "refId": "A",\n "units": "none",\n "valueHandler": "Text Only"\n },\n {\n "aggregation": "Last",\n "alias": "Total",\n "crit": 1,\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ceph_mon_quorum_status{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Total",\n "refId": "B",\n "units": "none",\n "valueHandler": "Text Only",\n "warn": 2\n },\n {\n "aggregation": "Last",\n "alias": "MONs out of Quorum",\n "crit": 1.6000000000000001,\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Warning / Critical",\n "displayType": "Annotation",\n "displayValueWithAlias": "Never",\n "expr": "count(ceph_mon_quorum_status{cluster=~\\"$cluster\\", }) - sum(ceph_mon_quorum_status{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "MONs out of Quorum",\n "range": true,\n "refId": "C",\n "units": "none",\n "valueHandler": "Number Threshold",\n "warn": 1.1000000000000001\n }\n ],\n "title": "Monitors",\n "transparent": false,\n "type": "stat"\n },\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 7\n },\n "id": 15,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "CLUSTER STATS",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "datasource": {\n "type": "datasource",\n "uid": "grafana"\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 0,\n "y": 8\n },\n "id": 16,\n "limit": 10,\n "onlyAlertsOnDashboard": true,\n "options": {\n "alertInstanceLabelFilter": "{alertname=~\\"^Ceph.+\\", cluster=~\\"$cluster\\", }",\n "alertName": "",\n "dashboardAlerts": false,\n "groupBy": [ ],\n "groupMode": "default",\n "maxItems": 20,\n "sortOrder": 1,\n "stateFilter": {\n "error": true,\n "firing": true,\n "noData": false,\n "normal": false,\n "pending": true\n },\n "viewMode": "list"\n },\n "show": "current",\n "sortOrder": 1,\n "stateFilter": [ ],\n "title": "Alerts",\n "type": "alertlist"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 40,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 0,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "percentage",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "#c0921f",\n "value": 75\n },\n {\n "color": "#E02F44",\n "value": 85\n }\n ]\n },\n "unit": "bytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "Total Capacity"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "red",\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Used"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "green",\n "mode": "fixed"\n }\n },\n {\n "id": "custom.thresholdsStyle",\n "value": {\n "mode": "dashed"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 8,\n "y": 8\n },\n "id": 17,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [\n "last"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "ceph_cluster_total_bytes{cluster=~\\"$cluster\\", }",\n "format": "time_series",\n "instant": false,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Total Capacity",\n "range": true,\n "refId": "A",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "ceph_cluster_total_used_bytes{cluster=~\\"$cluster\\", }",\n "format": "time_series",\n "instant": false,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Used",\n "range": true,\n "refId": "B",\n "step": 300\n }\n ],\n "title": "Capacity",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 85\n }\n ]\n },\n "unit": "bytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 16,\n "y": 8\n },\n "id": 18,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [\n "mean",\n "lastNotNull",\n "max",\n "min"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(irate(ceph_osd_op_w_in_bytes{cluster=~\\"$cluster\\", }[5m]))",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Write",\n "range": true,\n "refId": "A",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "sum(irate(ceph_osd_op_r_out_bytes{cluster=~\\"$cluster\\", }[5m]))",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Read",\n "range": true,\n "refId": "B",\n "step": 300\n }\n ],\n "title": "Cluster Throughput",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "bytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 0,\n "y": 16\n },\n "id": 19,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [\n "mean",\n "lastNotNull",\n "max",\n "min"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(irate(ceph_osd_op_w{cluster=~\\"$cluster\\", }[1m]))",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Write",\n "range": true,\n "refId": "A",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "sum(irate(ceph_osd_op_r{cluster=~\\"$cluster\\", }[1m]))",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Read",\n "range": true,\n "refId": "B",\n "step": 300\n }\n ],\n "title": "IOPS",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "bytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 8,\n "y": 16\n },\n "id": 20,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "right",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "(ceph_pool_bytes_used{cluster=~\\"$cluster\\", }) *on (pool_id) group_left(name)(ceph_pool_metadata{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Pool Used Bytes",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "bytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "rbd Stored"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "transparent",\n "mode": "fixed"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 16,\n "y": 16\n },\n "id": 21,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "table",\n "placement": "right",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "(ceph_pool_stored_raw{cluster=~\\"$cluster\\", }) *on (pool_id) group_left(name)(ceph_pool_metadata{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "hide": false,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Pool Used RAW Bytes",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 0,\n "y": 24\n },\n "id": 22,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "(ceph_pool_quota_objects{cluster=~\\"$cluster\\", }) *on (pool_id) group_left(name)(ceph_pool_metadata{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Pool Objects Quota",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "bytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 8,\n "y": 24\n },\n "id": 23,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "(ceph_pool_quota_bytes{cluster=~\\"$cluster\\", }) *on (pool_id) group_left(name)(ceph_pool_metadata{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Pool Quota Bytes",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 16,\n "y": 24\n },\n "id": 24,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "right",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "(ceph_pool_objects{cluster=~\\"$cluster\\", }) * on (pool_id) group_left(name)(ceph_pool_metadata{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "refId": "A"\n }\n ],\n "title": "Objects Per Pool",\n "type": "timeseries"\n },\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 31\n },\n "id": 25,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "OBJECTS",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 2,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byRegexp",\n "options": "/^Total.*$/"\n },\n "properties": [\n {\n "id": "custom.stacking",\n "value": {\n "group": false,\n "mode": "normal"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 12,\n "w": 6,\n "x": 0,\n "y": 32\n },\n "id": 26,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": false\n },\n "tooltip": {\n "mode": "multi",\n "sort": "asc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pool_objects{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Total",\n "range": true,\n "refId": "A",\n "step": 200\n }\n ],\n "title": "OSD Type Count",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 2,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byRegexp",\n "options": "/^Total.*$/"\n },\n "properties": [\n {\n "id": "custom.stacking",\n "value": {\n "group": false,\n "mode": "normal"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 12,\n "w": 8,\n "x": 6,\n "y": 32\n },\n "id": 27,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "right",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "asc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_active{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Active",\n "range": true,\n "refId": "A"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_clean{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Clean",\n "range": true,\n "refId": "B"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_peering{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Peering",\n "range": true,\n "refId": "C"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_degraded{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Degraded",\n "range": true,\n "refId": "D",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_stale{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Stale",\n "range": true,\n "refId": "E",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_unclean_pgs{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Unclean",\n "range": true,\n "refId": "F",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_undersized{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Undersized",\n "range": true,\n "refId": "G",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_incomplete{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Incomplete",\n "range": true,\n "refId": "H"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_forced_backfill{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Forced Backfill",\n "range": true,\n "refId": "I"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_forced_recovery{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Forced Recovery",\n "range": true,\n "refId": "J"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_creating{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Creating",\n "range": true,\n "refId": "K"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_wait_backfill{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Wait Backfill",\n "range": true,\n "refId": "L"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_deep{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Deep",\n "range": true,\n "refId": "M"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_scrubbing{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Scrubbing",\n "range": true,\n "refId": "N"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_recovering{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Recovering",\n "range": true,\n "refId": "O"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_repair{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Repair",\n "range": true,\n "refId": "P"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_down{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Down",\n "range": true,\n "refId": "Q"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_peered{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Peered",\n "range": true,\n "refId": "R"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_backfill{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Backfill",\n "range": true,\n "refId": "S"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_remapped{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Remapped",\n "range": true,\n "refId": "T"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_backfill_toofull{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Backfill Toofull",\n "range": true,\n "refId": "U"\n }\n ],\n "title": "PGs State",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 2,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byRegexp",\n "options": "/^Total.*$/"\n },\n "properties": [\n {\n "id": "custom.stacking",\n "value": {\n "group": false,\n "mode": "normal"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 6,\n "w": 10,\n "x": 14,\n "y": 32\n },\n "id": 28,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [\n "mean",\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "right",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "asc"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_degraded{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Degraded",\n "range": true,\n "refId": "A",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_stale{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Stale",\n "range": true,\n "refId": "B",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_undersized{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Undersized",\n "range": true,\n "refId": "C",\n "step": 300\n }\n ],\n "title": "Stuck PGs",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 2,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 6,\n "w": 10,\n "x": 14,\n "y": 38\n },\n "id": 29,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": false\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(irate(ceph_osd_recovery_ops{cluster=~\\"$cluster\\", }[$interval]))",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "OPS",\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Recovery Operations",\n "type": "timeseries"\n },\n {\n "collapse": false,\n "collapsed": true,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 44\n },\n "id": 30,\n "panels": [\n {\n "cards": {\n "cardPadding": null,\n "cardRound": null\n },\n "color": {\n "cardColor": "#b4ff00",\n "colorScale": "sqrt",\n "colorScheme": "interpolateOranges",\n "exponent": 0.5,\n "mode": "opacity"\n },\n "dataFormat": "timeseries",\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "scaleDistribution": {\n "type": "linear"\n }\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 0,\n "y": 42\n },\n "heatmap": { },\n "hideZeroBuckets": false,\n "highlightCards": true,\n "id": 31,\n "legend": {\n "show": true\n },\n "options": {\n "calculate": true,\n "calculation": {\n "yBuckets": {\n "mode": "count",\n "scale": {\n "log": 2,\n "type": "log"\n },\n "value": "1"\n }\n },\n "cellGap": 2,\n "cellValues": { },\n "color": {\n "exponent": 0.5,\n "fill": "#b4ff00",\n "mode": "opacity",\n "reverse": false,\n "scale": "exponential",\n "scheme": "Oranges",\n "steps": 128\n },\n "exemplars": {\n "color": "rgba(255,0,255,0.7)"\n },\n "filterValues": {\n "le": 1.0000000000000001e-09\n },\n "legend": {\n "show": true\n },\n "rowsFrame": {\n "layout": "auto"\n },\n "showValue": "never",\n "tooltip": {\n "show": true,\n "yHistogram": false\n },\n "yAxis": {\n "axisPlacement": "left",\n "min": "0",\n "reverse": false,\n "unit": "ms"\n }\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "ceph_osd_apply_latency_ms{cluster=~\\"$cluster\\", }",\n "format": "time_series",\n "instant": false,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "OSD Apply Latency Distribution",\n "tooltip": {\n "show": true,\n "showHistogram": false\n },\n "type": "heatmap",\n "xAxis": {\n "show": true\n },\n "xBucketNumber": null,\n "xBucketSize": "",\n "yAxis": {\n "decimals": null,\n "format": "ms",\n "logBase": 2,\n "max": null,\n "min": "0",\n "show": true,\n "splitFactor": 1\n },\n "yBucketBound": "auto",\n "yBucketNumber": null,\n "yBucketSize": 10\n },\n {\n "cards": {\n "cardPadding": null,\n "cardRound": null\n },\n "color": {\n "cardColor": "#65c5db",\n "colorScale": "sqrt",\n "colorScheme": "interpolateOranges",\n "exponent": 0.5,\n "mode": "opacity"\n },\n "dataFormat": "timeseries",\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "scaleDistribution": {\n "type": "linear"\n }\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 12,\n "y": 42\n },\n "heatmap": { },\n "hideZeroBuckets": false,\n "highlightCards": true,\n "id": 32,\n "legend": {\n "show": true\n },\n "options": {\n "calculate": true,\n "calculation": {\n "yBuckets": {\n "mode": "count",\n "scale": {\n "log": 2,\n "type": "log"\n }\n }\n },\n "cellGap": 2,\n "cellValues": { },\n "color": {\n "exponent": 0.5,\n "fill": "#65c5db",\n "mode": "opacity",\n "reverse": false,\n "scale": "exponential",\n "scheme": "Oranges",\n "steps": 128\n },\n "exemplars": {\n "color": "rgba(255,0,255,0.7)"\n },\n "filterValues": {\n "le": 1.0000000000000001e-09\n },\n "legend": {\n "show": true\n },\n "rowsFrame": {\n "layout": "auto"\n },\n "showValue": "never",\n "tooltip": {\n "show": true,\n "yHistogram": false\n },\n "yAxis": {\n "axisPlacement": "left",\n "min": "0",\n "reverse": false,\n "unit": "ms"\n }\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "ceph_osd_commit_latency_ms{cluster=~\\"$cluster\\", }",\n "format": "time_series",\n "instant": false,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "OSD Commit Latency Distribution",\n "tooltip": {\n "show": true,\n "showHistogram": false\n },\n "type": "heatmap",\n "xAxis": {\n "show": true\n },\n "xBucketNumber": null,\n "xBucketSize": "",\n "yAxis": {\n "decimals": null,\n "format": "ms",\n "logBase": 2,\n "max": null,\n "min": "0",\n "show": true,\n "splitFactor": 1\n },\n "yBucketBound": "auto",\n "yBucketNumber": null,\n "yBucketSize": 10\n },\n {\n "cards": {\n "cardPadding": null,\n "cardRound": null\n },\n "color": {\n "cardColor": "#806eb7",\n "colorScale": "sqrt",\n "colorScheme": "interpolateOranges",\n "exponent": 0.5,\n "mode": "opacity"\n },\n "dataFormat": "timeseries",\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "scaleDistribution": {\n "type": "linear"\n }\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 0,\n "y": 50\n },\n "heatmap": { },\n "hideZeroBuckets": false,\n "highlightCards": true,\n "id": 33,\n "legend": {\n "show": true\n },\n "options": {\n "calculate": true,\n "calculation": {\n "yBuckets": {\n "mode": "count",\n "scale": {\n "log": 2,\n "type": "log"\n }\n }\n },\n "cellGap": 2,\n "cellValues": { },\n "color": {\n "exponent": 0.5,\n "fill": "#806eb7",\n "mode": "opacity",\n "reverse": false,\n "scale": "exponential",\n "scheme": "Oranges",\n "steps": 128\n },\n "exemplars": {\n "color": "rgba(255,0,255,0.7)"\n },\n "filterValues": {\n "le": 1.0000000000000001e-09\n },\n "legend": {\n "show": true\n },\n "rowsFrame": {\n "layout": "auto"\n },\n "showValue": "never",\n "tooltip": {\n "show": true,\n "yHistogram": false\n },\n "yAxis": {\n "axisPlacement": "left",\n "decimals": 2,\n "min": "0",\n "reverse": false,\n "unit": "ms"\n }\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "rate(ceph_osd_op_r_latency_sum{cluster=~\\"$cluster\\", }[5m]) / rate(ceph_osd_op_r_latency_count{cluster=~\\"$cluster\\", }[5m]) >= 0",\n "format": "time_series",\n "instant": false,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "OSD Read Op Latency Distribution",\n "tooltip": {\n "show": true,\n "showHistogram": false\n },\n "type": "heatmap",\n "xAxis": {\n "show": true\n },\n "xBucketNumber": null,\n "xBucketSize": "",\n "yAxis": {\n "decimals": null,\n "format": "ms",\n "logBase": 2,\n "max": null,\n "min": "0",\n "show": true,\n "splitFactor": 1\n },\n "yBucketBound": "auto",\n "yBucketNumber": null,\n "yBucketSize": null\n },\n {\n "cards": {\n "cardPadding": null,\n "cardRound": null\n },\n "color": {\n "cardColor": "#f9934e",\n "colorScale": "sqrt",\n "colorScheme": "interpolateOranges",\n "exponent": 0.5,\n "mode": "opacity"\n },\n "dataFormat": "timeseries",\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "scaleDistribution": {\n "type": "linear"\n }\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 12,\n "y": 50\n },\n "heatmap": { },\n "hideZeroBuckets": false,\n "highlightCards": true,\n "id": 34,\n "legend": {\n "show": true\n },\n "options": {\n "calculate": true,\n "calculation": {\n "yBuckets": {\n "mode": "count",\n "scale": {\n "log": 2,\n "type": "log"\n }\n }\n },\n "cellGap": 2,\n "cellValues": { },\n "color": {\n "exponent": 0.5,\n "fill": "#f9934e",\n "mode": "opacity",\n "reverse": false,\n "scale": "exponential",\n "scheme": "Oranges",\n "steps": 128\n },\n "exemplars": {\n "color": "rgba(255,0,255,0.7)"\n },\n "filterValues": {\n "le": 1.0000000000000001e-09\n },\n "legend": {\n "show": true\n },\n "rowsFrame": {\n "layout": "auto"\n },\n "showValue": "never",\n "tooltip": {\n "show": true,\n "yHistogram": false\n },\n "yAxis": {\n "axisPlacement": "left",\n "decimals": 2,\n "min": "0",\n "reverse": false,\n "unit": "ms"\n }\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "rate(ceph_osd_op_w_latency_sum{cluster=~\\"$cluster\\", }[5m]) / rate(ceph_osd_op_w_latency_count{cluster=~\\"$cluster\\", }[5m]) >= 0",\n "format": "time_series",\n "instant": false,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "OSD Write Op Latency Distribution",\n "tooltip": {\n "show": true,\n "showHistogram": false\n },\n "type": "heatmap",\n "xAxis": {\n "show": true\n },\n "xBucketNumber": null,\n "xBucketSize": "",\n "yAxis": {\n "decimals": null,\n "format": "ms",\n "logBase": 2,\n "max": null,\n "min": "0",\n "show": true,\n "splitFactor": 1\n },\n "yBucketBound": "auto",\n "yBucketNumber": null,\n "yBucketSize": null\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "ms"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 12,\n "x": 0,\n "y": 58\n },\n "id": 35,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "avg(rate(ceph_osd_op_r_latency_sum{cluster=~\\"$cluster\\", }[5m]) / rate(ceph_osd_op_r_latency_count{cluster=~\\"$cluster\\", }[5m]) >= 0)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Read",\n "refId": "A"\n },\n {\n "datasource": "$datasource",\n "expr": "avg(rate(ceph_osd_op_w_latency_sum{cluster=~\\"$cluster\\", }[5m]) / rate(ceph_osd_op_w_latency_count{cluster=~\\"$cluster\\", }[5m]) >= 0)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Write",\n "refId": "B"\n }\n ],\n "title": "Recovery Operations",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "ms"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 12,\n "x": 12,\n "y": 58\n },\n "id": 36,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull",\n "max"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "avg(ceph_osd_apply_latency_ms{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "apply",\n "metric": "ceph_osd_perf_apply_latency_seconds",\n "refId": "A",\n "step": 4\n },\n {\n "datasource": "$datasource",\n "expr": "avg(ceph_osd_commit_latency_ms{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "commit",\n "metric": "ceph_osd_perf_commit_latency_seconds",\n "refId": "B",\n "step": 4\n }\n ],\n "title": "AVG OSD Apply + Commit Latency",\n "type": "timeseries"\n }\n ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "LATENCY",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "collapse": true,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 45\n },\n "id": 37,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "columns": [ ],\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "left",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": false,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "Time"\n },\n "properties": [\n {\n "id": "custom.hidden",\n "value": true\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 6,\n "w": 24,\n "x": 0,\n "y": 46\n },\n "id": 38,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "9.4.7",\n "styles": "",\n "targets": [\n {\n "datasource": "$datasource",\n "exemplar": false,\n "expr": "count by (ceph_version)(ceph_osd_metadata{cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "OSD Services",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": "$datasource",\n "exemplar": false,\n "expr": "count by (ceph_version)(ceph_mon_metadata{cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Mon Services",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": "$datasource",\n "exemplar": false,\n "expr": "count by (ceph_version)(ceph_mds_metadata{cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "MDS Services",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": "$datasource",\n "exemplar": false,\n "expr": "count by (ceph_version)(ceph_rgw_metadata{cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "RGW Services",\n "range": false,\n "refId": "D"\n },\n {\n "datasource": "$datasource",\n "exemplar": false,\n "expr": "count by (ceph_version)(ceph_mgr_metadata{cluster=~\\"$cluster\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "MGR Services",\n "range": false,\n "refId": "E"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Ceph Versions",\n "transformations": [\n {\n "id": "merge",\n "options": { }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": { },\n "indexByName": { },\n "renameByName": {\n "Time": "",\n "Value #A": "OSD Services",\n "Value #B": "Mon Services",\n "Value #C": "MDS Services",\n "Value #D": "RGW Services",\n "Value #E": "MGR Services",\n "ceph_version": "Ceph Version"\n }\n }\n }\n ],\n "type": "table"\n }\n ],\n "refresh": "1m",\n "rows": [ ],\n "schemaVersion": 38,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "auto": true,\n "auto_count": 10,\n "auto_min": "1m",\n "current": {\n "text": "$__auto_interval_interval",\n "value": "$__auto_interval_interval"\n },\n "hide": 0,\n "label": "Interval",\n "name": "interval",\n "options": [\n {\n "selected": true,\n "text": "auto",\n "value": "$__auto_interval_interval"\n },\n {\n "selected": false,\n "text": "5s",\n "value": "5s"\n },\n {\n "selected": false,\n "text": "10s",\n "value": "10s"\n },\n {\n "selected": false,\n "text": "30s",\n "value": "30s"\n },\n {\n "selected": false,\n "text": "1m",\n "value": "1m"\n },\n {\n "selected": false,\n "text": "10m",\n "value": "10m"\n },\n {\n "selected": false,\n "text": "30m",\n "value": "30m"\n },\n {\n "selected": false,\n "text": "1h",\n "value": "1h"\n },\n {\n "selected": false,\n "text": "6h",\n "value": "6h"\n },\n {\n "selected": false,\n "text": "12h",\n "value": "12h"\n },\n {\n "selected": false,\n "text": "1d",\n "value": "1d"\n },\n {\n "selected": false,\n "text": "7d",\n "value": "7d"\n },\n {\n "selected": false,\n "text": "14d",\n "value": "14d"\n },\n {\n "selected": false,\n "text": "30d",\n "value": "30d"\n }\n ],\n "query": "5s,10s,30s,1m,10m,30m,1h,6h,12h,1d,7d,14d,30d",\n "refresh": 2,\n "type": "interval",\n "valuelabels": { }\n }\n ]\n },\n "time": {\n "from": "now-6h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "Ceph Cluster - Advanced",\n "uid": "dn13KBeTv",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/osds-overview.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.0.0"\n },\n {\n "id": "grafana-piechart-panel",\n "name": "Pie Chart",\n "type": "panel",\n "version": "1.3.3"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n },\n {\n "id": "table",\n "name": "Table",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "ms"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "avg(\\n rate(ceph_osd_op_r_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_r_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval]) * 1000\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "AVG read",\n "refId": "A"\n },\n {\n "expr": "max(\\n rate(ceph_osd_op_r_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_r_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval]) * 1000\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "MAX read",\n "refId": "B"\n },\n {\n "expr": "quantile(0.95,\\n (\\n rate(ceph_osd_op_r_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_r_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval]) * 1000\\n )\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "@95%ile",\n "refId": "C"\n }\n ],\n "title": "OSD Read Latencies",\n "type": "timeseries"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "null",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "ceph_daemon"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "OSD ID"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Latency (ms)"\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 4,\n "x": 8,\n "y": 0\n },\n "id": 3,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "10.4.0",\n "styles": "",\n "targets": [\n {\n "expr": "topk(10,\\n (sort(\\n (\\n rate(ceph_osd_op_r_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_r_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n 1000\\n )\\n ))\\n)\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Highest READ Latencies",\n "transformations": [\n {\n "id": "merge",\n "options": {\n "reducers": [ ]\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time": true,\n "cluster": true\n },\n "includeByName": { },\n "indexByName": { },\n "renameByName": { }\n }\n }\n ],\n "type": "table"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "ms"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 12,\n "y": 0\n },\n "id": 4,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "avg(\\n rate(ceph_osd_op_w_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_w_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval]) * 1000\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "AVG write",\n "refId": "A"\n },\n {\n "expr": "max(\\n rate(ceph_osd_op_w_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_w_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval]) * 1000\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "MAX write",\n "refId": "B"\n },\n {\n "expr": "quantile(0.95,\\n (\\n rate(ceph_osd_op_w_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_w_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval]) * 1000\\n )\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "@95%ile write",\n "refId": "C"\n }\n ],\n "title": "OSD Write Latencies",\n "type": "timeseries"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "description": "This table shows the osd\'s that are delivering the 10 highest write latencies within the cluster",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "null",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "ceph_daemon"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "OSD ID"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Latency (ms)"\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "mappings",\n "value": [\n {\n "options": {\n "NaN": {\n "index": 0,\n "text": "0.00"\n }\n },\n "type": "value"\n }\n ]\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 4,\n "x": 20,\n "y": 0\n },\n "id": 5,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "10.4.0",\n "styles": "",\n "targets": [\n {\n "expr": "topk(10,\\n (sort(\\n (rate(ceph_osd_op_w_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_w_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n 1000)\\n ))\\n)\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Highest WRITE Latencies",\n "transformations": [\n {\n "id": "merge",\n "options": {\n "reducers": [ ]\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time": true,\n "cluster": true\n },\n "includeByName": { },\n "indexByName": { },\n "renameByName": { }\n }\n }\n ],\n "type": "table"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n }\n },\n "mappings": [ ]\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 4,\n "x": 0,\n "y": 8\n },\n "id": 6,\n "options": {\n "displayLabels": [\n "percent"\n ],\n "legend": {\n "calcs": [ ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "values": [\n "percent",\n "value"\n ]\n },\n "pieType": "pie",\n "reduceOptions": { },\n "tooltip": {\n "mode": "single",\n "sort": "none"\n }\n },\n "targets": [\n {\n "expr": "count by (device_class) (ceph_osd_metadata{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device_class}}",\n "refId": "A"\n }\n ],\n "title": "OSD Types Summary",\n "type": "piechart"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n }\n },\n "mappings": [ ]\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 4,\n "x": 4,\n "y": 8\n },\n "id": 7,\n "options": {\n "displayLabels": [\n "percent"\n ],\n "legend": {\n "calcs": [ ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "values": [\n "percent",\n "value"\n ]\n },\n "pieType": "pie",\n "reduceOptions": { },\n "tooltip": {\n "mode": "single",\n "sort": "none"\n }\n },\n "targets": [\n {\n "expr": "count(ceph_bluefs_wal_total_bytes{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "bluestore",\n "refId": "A"\n },\n {\n "expr": "absent(ceph_bluefs_wal_total_bytes{cluster=~\\"$cluster\\", }) * count(ceph_osd_metadata{cluster=~\\"$cluster\\", })",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "filestore",\n "refId": "B"\n }\n ],\n "title": "OSD Objectstore Types",\n "type": "piechart"\n },\n {\n "datasource": "$datasource",\n "description": "The pie chart shows the various OSD sizes used within the cluster",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n }\n },\n "mappings": [ ]\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 4,\n "x": 8,\n "y": 8\n },\n "id": 8,\n "options": {\n "displayLabels": [\n "percent"\n ],\n "legend": {\n "calcs": [ ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "values": [\n "percent",\n "value"\n ]\n },\n "pieType": "pie",\n "reduceOptions": { },\n "tooltip": {\n "mode": "single",\n "sort": "none"\n }\n },\n "targets": [\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster\\", } < 1099511627776)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<1TB",\n "refId": "A"\n },\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster\\", } >= 1099511627776 < 2199023255552)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<2TB",\n "refId": "B"\n },\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster\\", } >= 2199023255552 < 3298534883328)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<3TB",\n "refId": "C"\n },\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster\\", } >= 3298534883328 < 4398046511104)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<4TB",\n "refId": "D"\n },\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster\\", } >= 4398046511104 < 6597069766656)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<6TB",\n "refId": "E"\n },\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster\\", } >= 6597069766656 < 8796093022208)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<8TB",\n "refId": "F"\n },\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster\\", } >= 8796093022208 < 10995116277760)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<10TB",\n "refId": "G"\n },\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster\\", } >= 10995116277760 < 13194139533312)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<12TB",\n "refId": "H"\n },\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster\\", } >= 13194139533312)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<12TB+",\n "refId": "I"\n }\n ],\n "title": "OSD Size Summary",\n "type": "piechart"\n },\n {\n "aliasColors": { },\n "bars": true,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 12,\n "y": 8\n },\n "id": 9,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "ceph_osd_numpg{cluster=~\\"$cluster\\", }",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "PGs per OSD",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Distribution of PGs per OSD",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": 20,\n "mode": "histogram",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": "# of OSDs",\n "logBase": 1,\n "max": null,\n "min": "0",\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": "0",\n "show": true\n }\n ]\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": true,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "This gauge panel shows onode Hits ratio to help determine if increasing RAM per OSD could help improve the performance of the cluster",\n "format": "percentunit",\n "gauge": {\n "maxValue": 1,\n "minValue": 0,\n "show": true,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 8,\n "w": 4,\n "x": 20,\n "y": 8\n },\n "id": 10,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum(ceph_bluestore_onode_hits{cluster=~\\"$cluster\\", }) / (\\n sum(ceph_bluestore_onode_hits{cluster=~\\"$cluster\\", }) +\\n sum(ceph_bluestore_onode_misses{cluster=~\\"$cluster\\", })\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": ".75",\n "title": "OSD onode Hits Ratio",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 16\n },\n "id": 11,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "R/W Profile",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 24,\n "x": 0,\n "y": 17\n },\n "id": 12,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "round(sum(rate(ceph_pool_rd{cluster=~\\"$cluster\\", }[$__rate_interval])))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Reads",\n "refId": "A"\n },\n {\n "expr": "round(sum(rate(ceph_pool_wr{cluster=~\\"$cluster\\", }[$__rate_interval])))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Writes",\n "refId": "B"\n }\n ],\n "title": "Read/Write Profile",\n "type": "timeseries"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "description": "This table shows the 10 OSDs with the highest number of slow ops",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "null",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "ceph_daemon"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "OSD ID"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Slow Ops"\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 5,\n "x": 0,\n "y": 25\n },\n "id": 13,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "10.4.0",\n "styles": "",\n "targets": [\n {\n "expr": "topk(10,\\n (ceph_daemon_health_metrics{type=\\"SLOW_OPS\\", ceph_daemon=~\\"osd.*\\"})\\n)\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Top Slow Ops",\n "transformations": [\n {\n "id": "merge",\n "options": {\n "reducers": [ ]\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time": true,\n "__name__": true,\n "cluster": true,\n "instance": true,\n "job": true,\n "type": true\n },\n "includeByName": { },\n "indexByName": { },\n "renameByName": { }\n }\n }\n ],\n "type": "table"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "OSD Overview",\n "uid": "lo02I1Aiz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/rbd-overview.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.4.2"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n },\n {\n "id": "prometheus",\n "name": "Prometheus",\n "type": "datasource",\n "version": "5.0.0"\n },\n {\n "id": "table",\n "name": "Table",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "round(sum(rate(ceph_rbd_write_ops{cluster=~\\"$cluster\\", }[$__rate_interval])))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Writes",\n "refId": "A"\n },\n {\n "expr": "round(sum(rate(ceph_rbd_read_ops{cluster=~\\"$cluster\\", }[$__rate_interval])))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Reads",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "IOPS",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "Bps"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 8,\n "y": 0\n },\n "id": 3,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "round(sum(rate(ceph_rbd_write_bytes{cluster=~\\"$cluster\\", }[$__rate_interval])))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Write",\n "refId": "A"\n },\n {\n "expr": "round(sum(rate(ceph_rbd_read_bytes{cluster=~\\"$cluster\\", }[$__rate_interval])))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Read",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Throughput",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "Bps",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "ns"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 16,\n "y": 0\n },\n "id": 4,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "round(\\n sum(rate(ceph_rbd_write_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval])) /\\n sum(rate(ceph_rbd_write_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval]))\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Write",\n "refId": "A"\n },\n {\n "expr": "round(\\n sum(rate(ceph_rbd_read_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval])) /\\n sum(rate(ceph_rbd_read_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval]))\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Read",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Average Latency",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "ns",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "null",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "pool"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Pool"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "image"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Image"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "IOPS"\n },\n {\n "id": "unit",\n "value": "iops"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 0,\n "y": 7\n },\n "id": 5,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "10.4.0",\n "styles": "",\n "targets": [\n {\n "expr": "topk(10,\\n (\\n sort((\\n rate(ceph_rbd_write_ops{cluster=~\\"$cluster\\", }[$__rate_interval]) +\\n on (image, pool, namespace) rate(ceph_rbd_read_ops{cluster=~\\"$cluster\\", }[$__rate_interval])\\n ))\\n )\\n)\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Highest IOPS",\n "transformations": [\n {\n "id": "merge",\n "options": {\n "reducers": [ ]\n }\n }\n ],\n "type": "table"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "null",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "pool"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Pool"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "image"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Image"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Throughput"\n },\n {\n "id": "unit",\n "value": "Bps"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 8,\n "y": 7\n },\n "id": 6,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "10.4.0",\n "styles": "",\n "targets": [\n {\n "expr": "topk(10,\\n sort(\\n sum(\\n rate(ceph_rbd_read_bytes{cluster=~\\"$cluster\\", }[$__rate_interval]) +\\n rate(ceph_rbd_write_bytes{cluster=~\\"$cluster\\", }[$__rate_interval])\\n ) by (pool, image, namespace)\\n )\\n)\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Highest Throughput",\n "transformations": [\n {\n "id": "merge",\n "options": {\n "reducers": [ ]\n }\n }\n ],\n "type": "table"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "null",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "pool"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Pool"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "image"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Image"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Latency"\n },\n {\n "id": "unit",\n "value": "ns"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 16,\n "y": 7\n },\n "id": 7,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "10.4.0",\n "styles": "",\n "targets": [\n {\n "expr": "topk(10,\\n sum(\\n rate(ceph_rbd_write_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n clamp_min(rate(ceph_rbd_write_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval]), 1) +\\n rate(ceph_rbd_read_latency_sum{cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n clamp_min(rate(ceph_rbd_read_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval]), 1)\\n ) by (pool, image, namespace)\\n)\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Highest Latency",\n "transformations": [\n {\n "id": "merge",\n "options": {\n "reducers": [ ]\n }\n }\n ],\n "type": "table"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin",\n "overview"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "RBD Overview",\n "uid": "41FrpeUiz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/osd-device-details.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.3.2"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "OSD Performance",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "s"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 0,\n "y": 1\n },\n "id": 3,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "read",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "rate(ceph_osd_op_r_latency_sum{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_r_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval])\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "read",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_osd_op_w_latency_sum{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_w_latency_count{cluster=~\\"$cluster\\", }[$__rate_interval])\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "write",\n "refId": "B"\n }\n ],\n "title": "$osd Latency",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 6,\n "y": 1\n },\n "id": 4,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "Reads",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "rate(ceph_osd_op_r{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", }[$__rate_interval])",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Reads",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_osd_op_w{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", }[$__rate_interval])",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Writes",\n "refId": "B"\n }\n ],\n "title": "$osd R/W IOPS",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "bytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 12,\n "y": 1\n },\n "id": 5,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "Read Bytes",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "rate(ceph_osd_op_r_out_bytes{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", }[$__rate_interval])",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Read Bytes",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_osd_op_w_in_bytes{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", }[$__rate_interval])",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Write Bytes",\n "refId": "B"\n }\n ],\n "title": "$osd R/W Bytes",\n "type": "timeseries"\n },\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 10\n },\n "id": 6,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "Physical Device Performance",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "s"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 0,\n "y": 11\n },\n "id": 7,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*Reads/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "(\\n label_replace(\\n rate(node_disk_read_time_seconds_total[$__rate_interval]) /\\n rate(node_disk_reads_completed_total[$__rate_interval]),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n ) and on (instance, device) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", },\\n \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n )\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{instance}}/{{device}} Reads",\n "refId": "A"\n },\n {\n "expr": "(\\n label_replace(\\n rate(node_disk_write_time_seconds_total[$__rate_interval]) /\\n rate(node_disk_writes_completed_total[$__rate_interval]),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\") and on (instance, device)\\n label_replace(\\n label_replace(\\n ceph_disk_occupation_human{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", }, \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n )\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{instance}}/{{device}} Writes",\n "refId": "B"\n }\n ],\n "title": "Physical Device Latency for $osd",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 6,\n "y": 11\n },\n "id": 8,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*Reads/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "label_replace(\\n rate(node_disk_writes_completed_total[$__rate_interval]),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) and on (instance, device) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", },\\n \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}} on {{instance}} Writes",\n "refId": "A"\n },\n {\n "expr": "label_replace(\\n rate(node_disk_reads_completed_total[$__rate_interval]),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) and on (instance, device) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", },\\n \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}} on {{instance}} Reads",\n "refId": "B"\n }\n ],\n "title": "Physical Device R/W IOPS for $osd",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "Bps"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 12,\n "y": 11\n },\n "id": 9,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*Reads/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "label_replace(\\n rate(node_disk_read_bytes_total[$__rate_interval]), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) and on (instance, device) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", },\\n \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{instance}} {{device}} Reads",\n "refId": "A"\n },\n {\n "expr": "label_replace(\\n rate(node_disk_written_bytes_total[$__rate_interval]), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) and on (instance, device) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", },\\n \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{instance}} {{device}} Writes",\n "refId": "B"\n }\n ],\n "title": "Physical Device R/W Bytes for $osd",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "percentunit"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 18,\n "y": 11\n },\n "id": 10,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "label_replace(\\n rate(node_disk_io_time_seconds_total[$__rate_interval]),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) and on (instance, device) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster\\", }, \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}} on {{instance}}",\n "refId": "A"\n }\n ],\n "title": "Physical Device Util% for $osd",\n "type": "timeseries"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "OSD",\n "multi": false,\n "name": "osd",\n "options": [ ],\n "query": "label_values(ceph_osd_metadata{cluster=~\\"$cluster\\", }, ceph_daemon)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-3h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "OSD device details",\n "uid": "CrAHE0iZz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/cephfs-overview.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.3.2"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "MDS Performance",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Reads(-) / Writes (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "none"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 0,\n "y": 1\n },\n "id": 3,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*Reads/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "sum(rate(ceph_objecter_op_r{ceph_daemon=~\\"($mds_servers).*\\", cluster=~\\"$cluster\\", }[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Read Ops",\n "refId": "A"\n },\n {\n "expr": "sum(rate(ceph_objecter_op_w{ceph_daemon=~\\"($mds_servers).*\\", cluster=~\\"$cluster\\", }[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Write Ops",\n "refId": "B"\n }\n ],\n "title": "MDS Workload - $mds_servers",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Client Requests",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "none"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 12,\n "y": 1\n },\n "id": 4,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "ceph_mds_server_handle_client_request{ceph_daemon=~\\"($mds_servers).*\\", cluster=~\\"$cluster\\", }",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}}",\n "refId": "A"\n }\n ],\n "title": "Client Request Load - $mds_servers",\n "type": "timeseries"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": "MDS Server",\n "multi": false,\n "name": "mds_servers",\n "options": [ ],\n "query": "label_values(ceph_mds_inodes{cluster=~\\"$cluster\\", }, ceph_daemon)",\n "refresh": 1,\n "regex": "",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "MDS Performance",\n "uid": "tbO9LAiZz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/pool-detail.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.3.2"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n },\n {\n "id": "singlestat",\n "name": "Singlestat",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [ ],\n "panels": [\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": true,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "format": "percentunit",\n "gauge": {\n "maxValue": 1,\n "minValue": 0,\n "show": true,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 7,\n "w": 7,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": true\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "(ceph_pool_stored{cluster=~\\"$cluster\\", } / (ceph_pool_stored{cluster=~\\"$cluster\\", } + ceph_pool_max_avail{cluster=~\\"$cluster\\", })) *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{name=~\\"$pool_name\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": ".7,.8",\n "title": "Capacity used",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": 100,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Time till pool is full assuming the average fill rate of the last 6 hours",\n "format": "s",\n "gauge": {\n "maxValue": false,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 7,\n "w": 5,\n "x": 7,\n "y": 0\n },\n "id": 3,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": ""\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "(ceph_pool_max_avail{cluster=~\\"$cluster\\", } / deriv(ceph_pool_stored{cluster=~\\"$cluster\\", }[6h])) *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{name=~\\"$pool_name\\", cluster=~\\"$cluster\\", } > 0\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "current",\n "title": "Time till full",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": false\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Objects out(-) / in(+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "ops"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 12,\n "x": 12,\n "y": 0\n },\n "id": 4,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "deriv(ceph_pool_objects{cluster=~\\"$cluster\\", }[1m]) *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{name=~\\"$pool_name\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Objects per second",\n "refId": "A"\n }\n ],\n "title": "$pool_name Object Ingress/Egress",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "iops"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 12,\n "x": 0,\n "y": 7\n },\n "id": 5,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "reads",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "rate(ceph_pool_rd{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{name=~\\"$pool_name\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "reads",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_pool_wr{cluster=~\\"$cluster\\", }[$__rate_interval]) *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{name=~\\"$pool_name\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "writes",\n "refId": "B"\n }\n ],\n "title": "$pool_name Client IOPS",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "Bps"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 12,\n "x": 12,\n "y": 7\n },\n "id": 6,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "reads",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "rate(ceph_pool_rd_bytes{cluster=~\\"$cluster\\", }[$__rate_interval]) +\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{name=~\\"$pool_name\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "reads",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_pool_wr_bytes{cluster=~\\"$cluster\\", }[$__rate_interval]) +\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{name=~\\"$pool_name\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "writes",\n "refId": "B"\n }\n ],\n "title": "$pool_name Client Throughput",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Objects",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 12,\n "x": 0,\n "y": 14\n },\n "id": 7,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "ceph_pool_objects{cluster=~\\"$cluster\\", } *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{name=~\\"$pool_name\\", cluster=~\\"$cluster\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Number of Objects",\n "refId": "A"\n }\n ],\n "title": "$pool_name Objects",\n "type": "timeseries"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 22,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "Pool Name",\n "multi": false,\n "name": "pool_name",\n "options": [ ],\n "query": "label_values(ceph_pool_metadata{cluster=~\\"$cluster\\", }, name)",\n "refresh": 1,\n "regex": "",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "Ceph Pool Details",\n "uid": "-xyV8KCiz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/ceph-cluster.json': '{\n "__requires": [\n {\n "type": "grafana",\n "id": "grafana",\n "name": "Grafana",\n "version": "5.0.0"\n },\n {\n "type": "panel",\n "id": "graph",\n "name": "Graph",\n "version": "5.0.0"\n },\n {\n "type": "panel",\n "id": "heatmap",\n "name": "Heatmap",\n "version": "5.0.0"\n },\n {\n "type": "panel",\n "id": "singlestat",\n "name": "Singlestat",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": []\n },\n "description": "Ceph cluster overview",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "id": null,\n "iteration": 1525415495309,\n "links": [],\n "panels": [\n {\n "cacheTimeout": null,\n "colorBackground": true,\n "colorValue": false,\n "colors": [\n "rgba(50, 128, 45, 0.9)",\n "rgba(237, 129, 40, 0.9)",\n "rgb(255, 0, 0)"\n ],\n "datasource": "$datasource",\n "editable": false,\n "error": false,\n "format": "none",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 0,\n "y": 0\n },\n "hideTimeOverride": true,\n "id": 21,\n "interval": "1m",\n "links": [],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "span": 2,\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "ceph_health_status",\n "format": "time_series",\n "instant": true,\n "interval": "$interval",\n "intervalFactor": 1,\n "refId": "A",\n "step": 60\n }\n ],\n "thresholds": "1,2",\n "timeFrom": null,\n "title": "Health Status",\n "transparent": false,\n "type": "singlestat",\n "valueFontSize": "50%",\n "valueMaps": [\n {\n "op": "=",\n "text": "OK",\n "value": "0"\n },\n {\n "op": "=",\n "text": "WARN",\n "value": "1"\n },\n {\n "op": "=",\n "text": "ERR",\n "value": "2"\n }\n ],\n "valueName": "current"\n },\n {\n "colorMode": "Panel",\n "colors": {\n "crit": "rgb(255, 0, 0)",\n "disable": "rgba(128, 128, 128, 0.9)",\n "ok": "rgba(50, 128, 45, 0.9)",\n "warn": "rgba(237, 129, 40, 0.9)"\n },\n "cornerRadius": 0,\n "datasource": "$datasource",\n "displayName": "",\n "flipCard": false,\n "flipTime": 5,\n "fontFormat": "Regular",\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 6,\n "y": 0\n },\n "id": 43,\n "isAutoScrollOnOverflow": false,\n "isGrayOnNoData": false,\n "isHideAlertsOnDisable": false,\n "isIgnoreOKColors": false,\n "links": [],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n }\n },\n "targets": [\n {\n "aggregation": "Last",\n "alias": "All",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ceph_osd_metadata)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "All",\n "refId": "A",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "In",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_osd_in)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "In",\n "refId": "B",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "Out",\n "decimals": 2,\n "displayAliasType": "Warning / Critical",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_osd_in == bool 0)",\n "format": "time_series",\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Out",\n "refId": "C",\n "units": "none",\n "valueHandler": "Number Threshold",\n "warn": 1\n },\n {\n "aggregation": "Last",\n "alias": "Up",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_osd_up)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Up",\n "refId": "D",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "Down",\n "crit": 2,\n "decimals": 2,\n "displayAliasType": "Warning / Critical",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_osd_up == bool 0)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Down",\n "refId": "E",\n "units": "none",\n "valueHandler": "Number Threshold",\n "warn": 1\n }\n ],\n "title": "OSDs",\n "type": "stat",\n "fieldConfig": {\n "defaults": {\n "mappings": [],\n "thresholds": {\n "mode": "percentage",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "All"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Out"\n },\n "properties": [\n {\n "id": "thresholds",\n "value": {\n "mode": "percentage",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "orange",\n "value": 0.1\n },\n {\n "value": 10,\n "color": "red"\n }\n ]\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Down"\n },\n "properties": [\n {\n "id": "thresholds",\n "value": {\n "mode": "percentage",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "orange",\n "value": 0.1\n },\n {\n "value": 10,\n "color": "red"\n }\n ]\n }\n }\n ]\n }\n ]\n }\n },\n {\n "clusterName": "",\n "colorMode": "Panel",\n "colors": {\n "crit": "rgba(245, 54, 54, 0.9)",\n "disable": "rgba(128, 128, 128, 0.9)",\n "ok": "rgba(50, 128, 45, 0.9)",\n "warn": "rgba(237, 129, 40, 0.9)"\n },\n "cornerRadius": 1,\n "datasource": "$datasource",\n "displayName": "",\n "flipCard": false,\n "flipTime": 5,\n "fontFormat": "Regular",\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 12,\n "y": 0\n },\n "id": 41,\n "isAutoScrollOnOverflow": false,\n "isGrayOnNoData": false,\n "isHideAlertsOnDisable": false,\n "isIgnoreOKColors": false,\n "links": [],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n }\n },\n "targets": [\n {\n "aggregation": "Last",\n "alias": "In Quorum",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_mon_quorum_status)",\n "format": "time_series",\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "In Quorum",\n "refId": "A",\n "units": "none",\n "valueHandler": "Text Only"\n },\n {\n "aggregation": "Last",\n "alias": "Total",\n "crit": 1,\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ceph_mon_quorum_status)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Total",\n "refId": "B",\n "units": "none",\n "valueHandler": "Text Only",\n "warn": 2\n },\n {\n "aggregation": "Last",\n "alias": "MONs out of Quorum",\n "crit": 1.6,\n "decimals": 2,\n "displayAliasType": "Warning / Critical",\n "displayType": "Annotation",\n "displayValueWithAlias": "Never",\n "expr": "count(ceph_mon_quorum_status) - sum(ceph_mon_quorum_status)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "MONs out of Quorum",\n "refId": "C",\n "units": "none",\n "valueHandler": "Number Threshold",\n "warn": 1.1\n }\n ],\n "title": "Monitors",\n "type": "stat"\n },\n {\n "colorMode": "Panel",\n "colors": {\n "crit": "rgba(245, 54, 54, 0.9)",\n "disable": "rgba(128, 128, 128, 0.9)",\n "ok": "rgba(50, 128, 45, 0.9)",\n "warn": "rgba(237, 129, 40, 0.9)"\n },\n "cornerRadius": 1,\n "datasource": "$datasource",\n "displayName": "",\n "flipCard": false,\n "flipTime": 5,\n "fontFormat": "Regular",\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 18,\n "y": 0\n },\n "id": 68,\n "isAutoScrollOnOverflow": false,\n "isGrayOnNoData": false,\n "isHideAlertsOnDisable": false,\n "isIgnoreOKColors": false,\n "links": [],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n }\n },\n "targets": [\n {\n "aggregation": "Last",\n "alias": "Active",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ceph_mgr_status == 1) or vector(0)",\n "format": "time_series",\n "intervalFactor": 1,\n "instant": true,\n "legendFormat": "Active",\n "refId": "A",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "Standby",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ceph_mgr_status == 0) or vector(0)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Standby",\n "refId": "B",\n "units": "none",\n "valueHandler": "Number Threshold"\n }\n ],\n "title": "MGRs",\n "type": "stat"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "decimals": 2,\n "format": "percentunit",\n "gauge": {\n "maxValue": 1,\n "minValue": 0,\n "show": true,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 6,\n "w": 6,\n "x": 0,\n "y": 6\n },\n "id": 47,\n "interval": null,\n "links": [],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": true\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum(ceph_osd_stat_bytes_used)/sum(ceph_osd_stat_bytes)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Used",\n "refId": "A"\n }\n ],\n "thresholds": "0.7,0.8",\n "title": "Capacity used",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "aliasColors": {},\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 0,\n "gridPos": {\n "h": 6,\n "w": 9,\n "x": 6,\n "y": 6\n },\n "id": 53,\n "legend": {\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "show": true,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [],\n "nullPointMode": "null",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "seriesOverrides": [\n {\n "alias": "Active",\n "color": "#508642",\n "fill": 1,\n "stack": "A"\n },\n {\n "alias": "Total",\n "color": "#f9e2d2"\n },\n {\n "alias": "Degraded",\n "color": "#eab839"\n },\n {\n "alias": "Undersized",\n "color": "#f9934e"\n },\n {\n "alias": "Inconsistent",\n "color": "#e24d42"\n },\n {\n "alias": "Down",\n "color": "#bf1b00"\n },\n {\n "alias": "Inactive",\n "color": "#bf1b00",\n "fill": 4,\n "linewidth": 0,\n "stack": "A"\n }\n ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(ceph_pg_total)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Total",\n "refId": "A"\n },\n {\n "expr": "sum(ceph_pg_active)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Active",\n "refId": "B"\n },\n {\n "expr": "sum(ceph_pg_total - ceph_pg_active)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Inactive",\n "refId": "G"\n },\n {\n "expr": "sum(ceph_pg_undersized)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Undersized",\n "refId": "F"\n },\n {\n "expr": "sum(ceph_pg_degraded)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Degraded",\n "refId": "C"\n },\n {\n "expr": "sum(ceph_pg_inconsistent)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Inconsistent",\n "refId": "D"\n },\n {\n "expr": "sum(ceph_pg_down)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Down",\n "refId": "E"\n }\n ],\n "thresholds": [],\n "timeFrom": null,\n "timeShift": null,\n "title": "PG States",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": []\n },\n "yaxes": [\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": "0",\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": null,\n "show": false\n }\n ]\n },\n {\n "aliasColors": {},\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "ms"\n }\n },\n "fill": 0,\n "gridPos": {\n "h": 6,\n "w": 9,\n "x": 15,\n "y": 6\n },\n "id": 66,\n "legend": {\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "show": false,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [],\n "nullPointMode": "null",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "seriesOverrides": [\n {\n "alias": "Avg Apply Latency",\n "color": "#7eb26d"\n }\n ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "quantile(0.95, ceph_osd_apply_latency_ms)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Apply Latency P_95",\n "refId": "A"\n },\n {\n "expr": "quantile(0.95, ceph_osd_commit_latency_ms)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Commit Latency P_95",\n "refId": "B"\n },\n {\n "expr": "avg(ceph_osd_apply_latency_ms)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Avg Apply Latency",\n "refId": "C"\n },\n {\n "expr": "avg(ceph_osd_commit_latency_ms)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Avg Commit Latency",\n "refId": "D"\n }\n ],\n "thresholds": [],\n "timeFrom": null,\n "timeShift": null,\n "title": "OSD Latencies",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": []\n },\n "yaxes": [\n {\n "format": "ms",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": null,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": null,\n "show": true\n }\n ]\n },\n {\n "aliasColors": {},\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "Bps"\n }\n },\n "fill": 1,\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 0,\n "y": 9\n },\n "id": 45,\n "legend": {\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "show": true,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [],\n "nullPointMode": "null",\n "percentage": false,\n "pointradius": 0.5,\n "points": false,\n "renderer": "flot",\n "seriesOverrides": [\n {\n "alias": "Reads",\n "transform": "negative-Y"\n }\n ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(irate(ceph_osd_op_w_in_bytes[1m]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Writes",\n "refId": "A"\n },\n {\n "expr": "sum(irate(ceph_osd_op_r_out_bytes[1m]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Reads",\n "refId": "B"\n }\n ],\n "thresholds": [],\n "timeFrom": null,\n "timeShift": null,\n "title": "Cluster I/O",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": []\n },\n "yaxes": [\n {\n "format": "Bps",\n "label": "Read (-) / Write (+)",\n "logBase": 1,\n "max": null,\n "min": null,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": null,\n "show": false\n }\n ]\n },\n {\n "aliasColors": {},\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "Bps"\n }\n },\n "fill": 1,\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 12,\n "y": 9\n },\n "id": 62,\n "legend": {\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "show": false,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [],\n "nullPointMode": "null",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "seriesOverrides": [],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(deriv(ceph_pool_stored[1m]))",\n "format": "time_series",\n "intervalFactor": 1,\n "refId": "A"\n }\n ],\n "thresholds": [],\n "timeFrom": null,\n "timeShift": null,\n "title": "In-/Egress",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": []\n },\n "yaxes": [\n {\n "format": "Bps",\n "label": " Egress (-) / Ingress (+)",\n "logBase": 1,\n "max": null,\n "min": null,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": null,\n "show": false\n }\n ]\n },\n {\n "cards": {\n "cardPadding": null,\n "cardRound": 1\n },\n "color": {\n "cardColor": "rgb(0, 254, 255)",\n "colorScale": "sqrt",\n "colorScheme": "interpolateBlues",\n "exponent": 0.5,\n "min": null,\n "mode": "spectrum"\n },\n "dataFormat": "timeseries",\n "datasource": "$datasource",\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 0,\n "y": 15\n },\n "heatmap": {},\n "highlightCards": true,\n "id": 55,\n "legend": {\n "show": true\n },\n "links": [],\n "span": 12,\n "targets": [\n {\n "expr": "ceph_osd_stat_bytes_used / ceph_osd_stat_bytes",\n "format": "time_series",\n "interval": "1m",\n "intervalFactor": 1,\n "legendFormat": "Util (%)",\n "refId": "A",\n "step": 60\n }\n ],\n "timeFrom": null,\n "title": "OSD Capacity Utilization",\n "tooltip": {\n "show": true,\n "showHistogram": false\n },\n "type": "heatmap",\n "xAxis": {\n "show": true\n },\n "xBucketNumber": null,\n "xBucketSize": "",\n "yAxis": {\n "decimals": 2,\n "format": "percentunit",\n "logBase": 1,\n "max": null,\n "min": null,\n "show": true,\n "splitFactor": null\n },\n "yBucketNumber": null,\n "yBucketSize": null\n },\n {\n "cards": {\n "cardPadding": null,\n "cardRound": 1\n },\n "color": {\n "cardColor": "#b4ff00",\n "colorScale": "sqrt",\n "colorScheme": "interpolateBlues",\n "exponent": 0.5,\n "mode": "spectrum"\n },\n "dataFormat": "timeseries",\n "datasource": "$datasource",\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 6,\n "y": 15\n },\n "heatmap": {},\n "highlightCards": true,\n "id": 59,\n "legend": {\n "show": true\n },\n "links": [],\n "targets": [\n {\n "expr": "ceph_osd_numpg",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "#PGs",\n "refId": "A"\n }\n ],\n "title": "PGs per OSD",\n "tooltip": {\n "show": true,\n "showHistogram": false\n },\n "type": "heatmap",\n "xAxis": {\n "show": true\n },\n "xBucketNumber": null,\n "xBucketSize": "",\n "yAxis": {\n "decimals": null,\n "format": "none",\n "logBase": 1,\n "max": null,\n "min": null,\n "show": true,\n "splitFactor": null\n },\n "yBucketNumber": null,\n "yBucketSize": null\n },\n {\n "aliasColors": {},\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "ops"\n }\n },\n "fill": 0,\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 12,\n "y": 15\n },\n "id": 64,\n "legend": {\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "show": false,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [],\n "nullPointMode": "null",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "seriesOverrides": [],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(irate(ceph_osd_recovery_ops[1m]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Op/s",\n "refId": "A"\n }\n ],\n "thresholds": [],\n "timeFrom": null,\n "timeShift": null,\n "title": "Recovery Rate",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": []\n },\n "yaxes": [\n {\n "format": "ops",\n "label": "Recovery Ops/s",\n "logBase": 1,\n "max": null,\n "min": null,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": null,\n "show": true\n }\n ]\n }\n ],\n "refresh": "30s",\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph",\n "cluster"\n ],\n "templating": {\n "list": [\n {\n "hide": 0,\n "label": null,\n "name": "datasource",\n "options": [],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "auto": true,\n "auto_count": 10,\n "auto_min": "1m",\n "current": {\n "text": "auto",\n "value": "$__auto_interval_interval"\n },\n "datasource": null,\n "hide": 0,\n "includeAll": false,\n "label": "Interval",\n "multi": false,\n "name": "interval",\n "options": [\n {\n "selected": true,\n "text": "auto",\n "value": "$__auto_interval_interval"\n },\n {\n "selected": false,\n "text": "1m",\n "value": "1m"\n },\n {\n "selected": false,\n "text": "10m",\n "value": "10m"\n },\n {\n "selected": false,\n "text": "30m",\n "value": "30m"\n },\n {\n "selected": false,\n "text": "1h",\n "value": "1h"\n },\n {\n "selected": false,\n "text": "6h",\n "value": "6h"\n },\n {\n "selected": false,\n "text": "12h",\n "value": "12h"\n },\n {\n "selected": false,\n "text": "1d",\n "value": "1d"\n },\n {\n "selected": false,\n "text": "7d",\n "value": "7d"\n },\n {\n "selected": false,\n "text": "14d",\n "value": "14d"\n },\n {\n "selected": false,\n "text": "30d",\n "value": "30d"\n }\n ],\n "query": "1m,10m,30m,1h,6h,12h,1d,7d,14d,30d",\n "refresh": 2,\n "type": "interval"\n }\n ]\n },\n "time": {\n "from": "now-6h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "Ceph - Cluster",\n "version": 13\n}\n'}}} 2026-02-14 01:34:31,376 7ff87f1ede00 DEBUG Determined image: 'quay.io/ceph/grafana:10.4.0' 2026-02-14 01:34:31,390 7ff87f1ede00 INFO Reconfig daemon grafana.trial014 ... 2026-02-14 01:34:31,551 7ff87f1ede00 DEBUG firewalld.service is not enabled 2026-02-14 01:34:31,551 7ff87f1ede00 DEBUG Not possible to open ports <[3000]>. firewalld.service is not available 2026-02-14 01:34:32,131 7f6a18a0de00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.io/prometheus/prometheus:v2.51.0', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:34:32,149 7f6a18a0de00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'prometheus.trial014', 'image': '', 'deploy_arguments': [], 'params': {'tcp_ports': [9095], 'reconfig': True, 'allow_ptrace': True}, 'meta': {'service_name': 'prometheus', 'ports': [9095], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'files': {'prometheus.yml': '# This file is generated by cephadm.\nglobal:\n scrape_interval: 10s\n evaluation_interval: 10s\n external_labels:\n cluster: 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\nrule_files:\n - /etc/prometheus/alerting/*\n\nalerting:\n alertmanagers:\n - scheme: http\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=alertmanager\n\nscrape_configs:\n - job_name: \'ceph\'\n honor_labels: true\n relabel_configs:\n - source_labels: [__address__]\n target_label: cluster\n replacement: 29ce0a10-0945-11f1-b2be-d404e6e7d460\n - source_labels: [instance]\n target_label: instance\n replacement: \'ceph_cluster\'\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=mgr-prometheus\n\n - job_name: \'node\'\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=node-exporter\n relabel_configs:\n - source_labels: [__address__]\n target_label: cluster\n replacement: 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\n\n - job_name: \'ceph-exporter\'\n honor_labels: true\n relabel_configs:\n - source_labels: [__address__]\n target_label: cluster\n replacement: 29ce0a10-0945-11f1-b2be-d404e6e7d460\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=ceph-exporter\n\n - job_name: \'nvmeof\'\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=nvmeof\n\n - job_name: \'nfs\'\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=nfs\n\n - job_name: \'federate\'\n scrape_interval: 15s\n honor_labels: true\n metrics_path: \'/federate\'\n params:\n \'match[]\':\n - \'{job="ceph"}\'\n - \'{job="node"}\'\n - \'{job="haproxy"}\'\n - \'{job="ceph-exporter"}\'\n static_configs:\n - targets: []\n', '/etc/prometheus/alerting/ceph_alerts.yml': 'groups:\n - name: "cluster health"\n rules:\n - alert: "CephHealthError"\n annotations:\n description: "The cluster state has been HEALTH_ERROR for more than 5 minutes. Please check \'ceph health detail\' for more information."\n summary: "Ceph is in the ERROR state"\n expr: "ceph_health_status == 2"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.2.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephHealthWarning"\n annotations:\n description: "The cluster state has been HEALTH_WARN for more than 15 minutes. Please check \'ceph health detail\' for more information."\n summary: "Ceph is in the WARNING state"\n expr: "ceph_health_status == 1"\n for: "15m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "mon"\n rules:\n - alert: "CephMonDownQuorumAtRisk"\n annotations:\n description: "{{ $min := query \\"floor(count(ceph_mon_metadata) / 2) + 1\\" | first | value }}Quorum requires a majority of monitors (x {{ $min }}) to be active. Without quorum the cluster will become inoperable, affecting all services and connected clients. The following monitors are down: {{- range query \\"(ceph_mon_quorum_status == 0) + on(ceph_daemon) group_left(hostname) (ceph_mon_metadata * 0)\\" }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-down"\n summary: "Monitor quorum is at risk"\n expr: |\n (\n (ceph_health_detail{name="MON_DOWN"} == 1) * on() (\n count(ceph_mon_quorum_status == 1) == bool (floor(count(ceph_mon_metadata) / 2) + 1)\n )\n ) == 1\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.3.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephMonDown"\n annotations:\n description: |\n {{ $down := query "count(ceph_mon_quorum_status == 0)" | first | value }}{{ $s := "" }}{{ if gt $down 1.0 }}{{ $s = "s" }}{{ end }}You have {{ $down }} monitor{{ $s }} down. Quorum is still intact, but the loss of an additional monitor will make your cluster inoperable. The following monitors are down: {{- range query "(ceph_mon_quorum_status == 0) + on(ceph_daemon) group_left(hostname) (ceph_mon_metadata * 0)" }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-down"\n summary: "One or more monitors down"\n expr: |\n count(ceph_mon_quorum_status == 0) <= (count(ceph_mon_metadata) - floor(count(ceph_mon_metadata) / 2) + 1)\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephMonDiskspaceCritical"\n annotations:\n description: "The free space available to a monitor\'s store is critically low. You should increase the space available to the monitor(s). The default directory is /var/lib/ceph/mon-*/data/store.db on traditional deployments, and /var/lib/rook/mon-*/data/store.db on the mon pod\'s worker node for Rook. Look for old, rotated versions of *.log and MANIFEST*. Do NOT touch any *.sst files. Also check any other directories under /var/lib/rook and other directories on the same filesystem, often /var/log and /var/tmp are culprits. Your monitor hosts are; {{- range query \\"ceph_mon_metadata\\"}} - {{ .Labels.hostname }} {{- end }}"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-disk-crit"\n summary: "Filesystem space on at least one monitor is critically low"\n expr: "ceph_health_detail{name=\\"MON_DISK_CRIT\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.3.2"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephMonDiskspaceLow"\n annotations:\n description: "The space available to a monitor\'s store is approaching full (>70% is the default). You should increase the space available to the monitor(s). The default directory is /var/lib/ceph/mon-*/data/store.db on traditional deployments, and /var/lib/rook/mon-*/data/store.db on the mon pod\'s worker node for Rook. Look for old, rotated versions of *.log and MANIFEST*. Do NOT touch any *.sst files. Also check any other directories under /var/lib/rook and other directories on the same filesystem, often /var/log and /var/tmp are culprits. Your monitor hosts are; {{- range query \\"ceph_mon_metadata\\"}} - {{ .Labels.hostname }} {{- end }}"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-disk-low"\n summary: "Drive space on at least one monitor is approaching full"\n expr: "ceph_health_detail{name=\\"MON_DISK_LOW\\"} == 1"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephMonClockSkew"\n annotations:\n description: "Ceph monitors rely on closely synchronized time to maintain quorum and cluster consistency. This event indicates that the time on at least one mon has drifted too far from the lead mon. Review cluster status with ceph -s. This will show which monitors are affected. Check the time sync status on each monitor host with \'ceph time-sync-status\' and the state and peers of your ntpd or chrony daemon."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-clock-skew"\n summary: "Clock skew detected among monitors"\n expr: "ceph_health_detail{name=\\"MON_CLOCK_SKEW\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "osd"\n rules:\n - alert: "CephOSDDownHigh"\n annotations:\n description: "{{ $value | humanize }}% or {{ with query \\"count(ceph_osd_up == 0)\\" }}{{ . | first | value }}{{ end }} of {{ with query \\"count(ceph_osd_up)\\" }}{{ . | first | value }}{{ end }} OSDs are down (>= 10%). The following OSDs are down: {{- range query \\"(ceph_osd_up * on(ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0\\" }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}"\n summary: "More than 10% of OSDs are down"\n expr: "count(ceph_osd_up == 0) / count(ceph_osd_up) * 100 >= 10"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephOSDHostDown"\n annotations:\n description: "The following OSDs are down: {{- range query \\"(ceph_osd_up * on(ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0\\" }} - {{ .Labels.hostname }} : {{ .Labels.ceph_daemon }} {{- end }}"\n summary: "An OSD host is offline"\n expr: "ceph_health_detail{name=\\"OSD_HOST_DOWN\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.8"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDDown"\n annotations:\n description: |\n {{ $num := query "count(ceph_osd_up == 0)" | first | value }}{{ $s := "" }}{{ if gt $num 1.0 }}{{ $s = "s" }}{{ end }}{{ $num }} OSD{{ $s }} down for over 5mins. The following OSD{{ $s }} {{ if eq $s "" }}is{{ else }}are{{ end }} down: {{- range query "(ceph_osd_up * on(ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0"}} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-down"\n summary: "An OSD has been marked down"\n expr: "ceph_health_detail{name=\\"OSD_DOWN\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.2"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDNearFull"\n annotations:\n description: "One or more OSDs have reached the NEARFULL threshold. Use \'ceph health detail\' and \'ceph osd df\' to identify the problem. To resolve, add capacity to the affected OSD\'s failure domain, restore down/out OSDs, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-nearfull"\n summary: "OSD(s) running low on free space (NEARFULL)"\n expr: "ceph_health_detail{name=\\"OSD_NEARFULL\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.3"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDFull"\n annotations:\n description: "An OSD has reached the FULL threshold. Writes to pools that share the affected OSD will be blocked. Use \'ceph health detail\' and \'ceph osd df\' to identify the problem. To resolve, add capacity to the affected OSD\'s failure domain, restore down/out OSDs, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-full"\n summary: "OSD full, writes blocked"\n expr: "ceph_health_detail{name=\\"OSD_FULL\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.6"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephOSDBackfillFull"\n annotations:\n description: "An OSD has reached the BACKFILL FULL threshold. This will prevent rebalance operations from completing. Use \'ceph health detail\' and \'ceph osd df\' to identify the problem. To resolve, add capacity to the affected OSD\'s failure domain, restore down/out OSDs, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-backfillfull"\n summary: "OSD(s) too full for backfill operations"\n expr: "ceph_health_detail{name=\\"OSD_BACKFILLFULL\\"} > 0"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDTooManyRepairs"\n annotations:\n description: "Reads from an OSD have used a secondary PG to return data to the client, indicating a potential failing drive."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-too-many-repairs"\n summary: "OSD reports a high number of read errors"\n expr: "ceph_health_detail{name=\\"OSD_TOO_MANY_REPAIRS\\"} == 1"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDTimeoutsPublicNetwork"\n annotations:\n description: "OSD heartbeats on the cluster\'s \'public\' network (frontend) are running slow. Investigate the network for latency or loss issues. Use \'ceph health detail\' to show the affected OSDs."\n summary: "Network issues delaying OSD heartbeats (public network)"\n expr: "ceph_health_detail{name=\\"OSD_SLOW_PING_TIME_FRONT\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDTimeoutsClusterNetwork"\n annotations:\n description: "OSD heartbeats on the cluster\'s \'cluster\' network (backend) are slow. Investigate the network for latency issues on this subnet. Use \'ceph health detail\' to show the affected OSDs."\n summary: "Network issues delaying OSD heartbeats (cluster network)"\n expr: "ceph_health_detail{name=\\"OSD_SLOW_PING_TIME_BACK\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDInternalDiskSizeMismatch"\n annotations:\n description: "One or more OSDs have an internal inconsistency between metadata and the size of the device. This could lead to the OSD(s) crashing in future. You should redeploy the affected OSDs."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#bluestore-disk-size-mismatch"\n summary: "OSD size inconsistency error"\n expr: "ceph_health_detail{name=\\"BLUESTORE_DISK_SIZE_MISMATCH\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephDeviceFailurePredicted"\n annotations:\n description: "The device health module has determined that one or more devices will fail soon. To review device status use \'ceph device ls\'. To show a specific device use \'ceph device info \'. Mark the OSD out so that data may migrate to other OSDs. Once the OSD has drained, destroy the OSD, replace the device, and redeploy the OSD."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#id2"\n summary: "Device(s) predicted to fail soon"\n expr: "ceph_health_detail{name=\\"DEVICE_HEALTH\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephDeviceFailurePredictionTooHigh"\n annotations:\n description: "The device health module has determined that devices predicted to fail can not be remediated automatically, since too many OSDs would be removed from the cluster to ensure performance and availability. Prevent data integrity issues by adding new OSDs so that data may be relocated."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#device-health-toomany"\n summary: "Too many devices are predicted to fail, unable to resolve"\n expr: "ceph_health_detail{name=\\"DEVICE_HEALTH_TOOMANY\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.7"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephDeviceFailureRelocationIncomplete"\n annotations:\n description: "The device health module has determined that one or more devices will fail soon, but the normal process of relocating the data on the device to other OSDs in the cluster is blocked. \\nEnsure that the cluster has available free space. It may be necessary to add capacity to the cluster to allow data from the failing device to successfully migrate, or to enable the balancer."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#device-health-in-use"\n summary: "Device failure is predicted, but unable to relocate data"\n expr: "ceph_health_detail{name=\\"DEVICE_HEALTH_IN_USE\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDFlapping"\n annotations:\n description: "OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked down and back up {{ $value | humanize }} times once a minute for 5 minutes. This may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster network, or the public network if no cluster network is deployed. Check the network stats on the listed host(s)."\n documentation: "https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds"\n summary: "Network issues are causing OSDs to flap (mark each other down)"\n expr: "(rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata) * 60 > 1"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.4"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDReadErrors"\n annotations:\n description: "An OSD has encountered read errors, but the OSD has recovered by retrying the reads. This may indicate an issue with hardware or the kernel."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#bluestore-spurious-read-errors"\n summary: "Device read errors detected"\n expr: "ceph_health_detail{name=\\"BLUESTORE_SPURIOUS_READ_ERRORS\\"} == 1"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPGImbalance"\n annotations:\n description: "OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} deviates by more than 30% from average PG count."\n summary: "PGs are not balanced across OSDs"\n expr: |\n abs(\n ((ceph_osd_numpg > 0) - on (job) group_left avg(ceph_osd_numpg > 0) by (job)) /\n on (job) group_left avg(ceph_osd_numpg > 0) by (job)\n ) * on (ceph_daemon) group_left(hostname) ceph_osd_metadata > 0.30\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.5"\n severity: "warning"\n type: "ceph_default"\n - name: "mds"\n rules:\n - alert: "CephFilesystemDamaged"\n annotations:\n description: "Filesystem metadata has been corrupted. Data may be inaccessible. Analyze metrics from the MDS daemon admin socket, or escalate to support."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages#cephfs-health-messages"\n summary: "CephFS filesystem is damaged."\n expr: "ceph_health_detail{name=\\"MDS_DAMAGE\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephFilesystemOffline"\n annotations:\n description: "All MDS ranks are unavailable. The MDS daemons managing metadata are down, rendering the filesystem offline."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-all-down"\n summary: "CephFS filesystem is offline"\n expr: "ceph_health_detail{name=\\"MDS_ALL_DOWN\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.3"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephFilesystemDegraded"\n annotations:\n description: "One or more metadata daemons (MDS ranks) are failed or in a damaged state. At best the filesystem is partially available, at worst the filesystem is completely unusable."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#fs-degraded"\n summary: "CephFS filesystem is degraded"\n expr: "ceph_health_detail{name=\\"FS_DEGRADED\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.4"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephFilesystemMDSRanksLow"\n annotations:\n description: "The filesystem\'s \'max_mds\' setting defines the number of MDS ranks in the filesystem. The current number of active MDS daemons is less than this value."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-up-less-than-max"\n summary: "Ceph MDS daemon count is lower than configured"\n expr: "ceph_health_detail{name=\\"MDS_UP_LESS_THAN_MAX\\"} > 0"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephFilesystemInsufficientStandby"\n annotations:\n description: "The minimum number of standby daemons required by standby_count_wanted is less than the current number of standby daemons. Adjust the standby count or increase the number of MDS daemons."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-insufficient-standby"\n summary: "Ceph filesystem standby daemons too few"\n expr: "ceph_health_detail{name=\\"MDS_INSUFFICIENT_STANDBY\\"} > 0"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephFilesystemFailureNoStandby"\n annotations:\n description: "An MDS daemon has failed, leaving only one active rank and no available standby. Investigate the cause of the failure or add a standby MDS."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#fs-with-failed-mds"\n summary: "MDS daemon failed, no further standby available"\n expr: "ceph_health_detail{name=\\"FS_WITH_FAILED_MDS\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.5"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephFilesystemReadOnly"\n annotations:\n description: "The filesystem has switched to READ ONLY due to an unexpected error when writing to the metadata pool. Either analyze the output from the MDS daemon admin socket, or escalate to support."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages#cephfs-health-messages"\n summary: "CephFS filesystem in read only mode due to write error(s)"\n expr: "ceph_health_detail{name=\\"MDS_HEALTH_READ_ONLY\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.2"\n severity: "critical"\n type: "ceph_default"\n - name: "mgr"\n rules:\n - alert: "CephMgrModuleCrash"\n annotations:\n description: "One or more mgr modules have crashed and have yet to be acknowledged by an administrator. A crashed module may impact functionality within the cluster. Use the \'ceph crash\' command to determine which module has failed, and archive it to acknowledge the failure."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#recent-mgr-module-crash"\n summary: "A manager module has recently crashed"\n expr: "ceph_health_detail{name=\\"RECENT_MGR_MODULE_CRASH\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.6.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephMgrPrometheusModuleInactive"\n annotations:\n description: "The mgr/prometheus module at {{ $labels.instance }} is unreachable. This could mean that the module has been disabled or the mgr daemon itself is down. Without the mgr/prometheus module metrics and alerts will no longer function. Open a shell to an admin node or toolbox pod and use \'ceph -s\' to to determine whether the mgr is active. If the mgr is not active, restart it, otherwise you can determine module status with \'ceph mgr module ls\'. If it is not listed as enabled, enable it with \'ceph mgr module enable prometheus\'."\n summary: "The mgr/prometheus module is not available"\n expr: "up{job=\\"ceph\\"} == 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.6.2"\n severity: "critical"\n type: "ceph_default"\n - name: "pgs"\n rules:\n - alert: "CephPGsInactive"\n annotations:\n description: "{{ $value }} PGs have been inactive for more than 5 minutes in pool {{ $labels.name }}. Inactive placement groups are not able to serve read/write requests."\n summary: "One or more placement groups are inactive"\n expr: "ceph_pool_metadata * on(pool_id,instance) group_left() (ceph_pg_total - ceph_pg_active) > 0"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGsUnclean"\n annotations:\n description: "{{ $value }} PGs have been unclean for more than 15 minutes in pool {{ $labels.name }}. Unclean PGs have not recovered from a previous failure."\n summary: "One or more placement groups are marked unclean"\n expr: "ceph_pool_metadata * on(pool_id,instance) group_left() (ceph_pg_total - ceph_pg_clean) > 0"\n for: "15m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.2"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPGsDamaged"\n annotations:\n description: "During data consistency checks (scrub), at least one PG has been flagged as being damaged or inconsistent. Check to see which PG is affected, and attempt a manual repair if necessary. To list problematic placement groups, use \'rados list-inconsistent-pg \'. To repair PGs use the \'ceph pg repair \' command."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-damaged"\n summary: "Placement group damaged, manual intervention needed"\n expr: "ceph_health_detail{name=~\\"PG_DAMAGED|OSD_SCRUB_ERRORS\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.4"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGRecoveryAtRisk"\n annotations:\n description: "Data redundancy is at risk since one or more OSDs are at or above the \'full\' threshold. Add more capacity to the cluster, restore down/out OSDs, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-recovery-full"\n summary: "OSDs are too full for recovery"\n expr: "ceph_health_detail{name=\\"PG_RECOVERY_FULL\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.5"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGUnavailableBlockingIO"\n annotations:\n description: "Data availability is reduced, impacting the cluster\'s ability to service I/O. One or more placement groups (PGs) are in a state that blocks I/O."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-availability"\n summary: "PG is unavailable, blocking I/O"\n expr: "((ceph_health_detail{name=\\"PG_AVAILABILITY\\"} == 1) - scalar(ceph_health_detail{name=\\"OSD_DOWN\\"})) == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.3"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGBackfillAtRisk"\n annotations:\n description: "Data redundancy may be at risk due to lack of free space within the cluster. One or more OSDs have reached the \'backfillfull\' threshold. Add more capacity, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-backfill-full"\n summary: "Backfill operations are blocked due to lack of free space"\n expr: "ceph_health_detail{name=\\"PG_BACKFILL_FULL\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.6"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGNotScrubbed"\n annotations:\n description: "One or more PGs have not been scrubbed recently. Scrubs check metadata integrity, protecting against bit-rot. They check that metadata is consistent across data replicas. When PGs miss their scrub interval, it may indicate that the scrub window is too small, or PGs were not in a \'clean\' state during the scrub window. You can manually initiate a scrub with: ceph pg scrub "\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-not-scrubbed"\n summary: "Placement group(s) have not been scrubbed"\n expr: "ceph_health_detail{name=\\"PG_NOT_SCRUBBED\\"} == 1"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPGsHighPerOSD"\n annotations:\n description: "The number of placement groups per OSD is too high (exceeds the mon_max_pg_per_osd setting).\\n Check that the pg_autoscaler has not been disabled for any pools with \'ceph osd pool autoscale-status\', and that the profile selected is appropriate. You may also adjust the target_size_ratio of a pool to guide the autoscaler based on the expected relative size of the pool (\'ceph osd pool set cephfs.cephfs.meta target_size_ratio .1\') or set the pg_autoscaler mode to \'warn\' and adjust pg_num appropriately for one or more pools."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks/#too-many-pgs"\n summary: "Placement groups per OSD is too high"\n expr: "ceph_health_detail{name=\\"TOO_MANY_PGS\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPGNotDeepScrubbed"\n annotations:\n description: "One or more PGs have not been deep scrubbed recently. Deep scrubs protect against bit-rot. They compare data replicas to ensure consistency. When PGs miss their deep scrub interval, it may indicate that the window is too small or PGs were not in a \'clean\' state during the deep-scrub window."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-not-deep-scrubbed"\n summary: "Placement group(s) have not been deep scrubbed"\n expr: "ceph_health_detail{name=\\"PG_NOT_DEEP_SCRUBBED\\"} == 1"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "nodes"\n rules:\n - alert: "CephNodeRootFilesystemFull"\n annotations:\n description: "Root volume is dangerously full: {{ $value | humanize }}% free."\n summary: "Root filesystem is dangerously full"\n expr: "node_filesystem_avail_bytes{mountpoint=\\"/\\"} / node_filesystem_size_bytes{mountpoint=\\"/\\"} * 100 < 5"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.8.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephNodeNetworkPacketDrops"\n annotations:\n description: "Node {{ $labels.instance }} experiences packet drop > 0.5% or > 10 packets/s on interface {{ $labels.device }}."\n summary: "One or more NICs reports packet drops"\n expr: |\n (\n rate(node_network_receive_drop_total{device!="lo"}[1m]) +\n rate(node_network_transmit_drop_total{device!="lo"}[1m])\n ) / (\n rate(node_network_receive_packets_total{device!="lo"}[1m]) +\n rate(node_network_transmit_packets_total{device!="lo"}[1m])\n ) >= 0.0050000000000000001 and (\n rate(node_network_receive_drop_total{device!="lo"}[1m]) +\n rate(node_network_transmit_drop_total{device!="lo"}[1m])\n ) >= 10\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.8.2"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephNodeNetworkPacketErrors"\n annotations:\n description: "Node {{ $labels.instance }} experiences packet errors > 0.01% or > 10 packets/s on interface {{ $labels.device }}."\n summary: "One or more NICs reports packet errors"\n expr: |\n (\n rate(node_network_receive_errs_total{device!="lo"}[1m]) +\n rate(node_network_transmit_errs_total{device!="lo"}[1m])\n ) / (\n rate(node_network_receive_packets_total{device!="lo"}[1m]) +\n rate(node_network_transmit_packets_total{device!="lo"}[1m])\n ) >= 0.0001 or (\n rate(node_network_receive_errs_total{device!="lo"}[1m]) +\n rate(node_network_transmit_errs_total{device!="lo"}[1m])\n ) >= 10\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.8.3"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephNodeNetworkBondDegraded"\n annotations:\n description: "Bond {{ $labels.master }} is degraded on Node {{ $labels.instance }}."\n summary: "Degraded Bond on Node {{ $labels.instance }}"\n expr: |\n node_bonding_slaves - node_bonding_active != 0\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephNodeDiskspaceWarning"\n annotations:\n description: "Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will be full in less than 5 days based on the 48 hour trailing fill rate."\n summary: "Host filesystem free space is getting low"\n expr: "predict_linear(node_filesystem_free_bytes{device=~\\"/.*\\"}[2d], 3600 * 24 * 5) *on(instance) group_left(nodename) node_uname_info < 0"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.8.4"\n severity: "warning"\n type: "ceph_default"\n - alert: CephNodeInconsistentMTU\n expr: |\n node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)\n != on (cluster, device) group_left\n quantile by (cluster, device) (\n 0.5, node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)\n )\n labels:\n severity: warning\n type: ceph_default\n annotations:\n summary: "Node {{ $labels.instance }} has inconsistent MTU settings in cluster {{ $labels.cluster }}"\n description: "Network interface {{ $labels.device }} on node {{ $labels.instance }} has MTU {{ $value }} which differs from the cluster median."\n impact: |\n - May cause packet fragmentation or packet drops\n - Risk of degraded cluster communication and performance\n - Potential instability in services relying on consistent networking (e.g., Ceph, Kubernetes)\n fix: |\n - Check the MTU of interface `{{ $labels.device }}` on node `{{ $labels.instance }}`:\n ip link show {{ $labels.device }}\n\n - Find the median MTU value across the cluster by running this PromQL query in Prometheus:\n quantile by (cluster, device) (0.5, node_network_mtu_bytes * (node_network_up{device!="lo"} > 0))\n\n - Standardize MTU across all nodes to match the median (commonly 1500 or 9000):\n ip link set dev {{ $labels.device }} mtu \n\n - Make MTU setting persistent:\n - RHEL/CentOS: edit `/etc/sysconfig/network-scripts/ifcfg-`\n - Debian/Ubuntu: edit `/etc/netplan/*.yaml` and apply with `netplan apply`\n\n - Restart the affected interface or node if required.\n - name: "pools"\n rules:\n - alert: "CephPoolGrowthWarning"\n annotations:\n description: "Pool \'{{ $labels.name }}\' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours."\n summary: "Pool growth rate may soon exceed capacity"\n expr: "(predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id, instance) group_right() ceph_pool_metadata) >= 95"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.9.2"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPoolBackfillFull"\n annotations:\n description: "A pool is approaching the near full threshold, which will prevent recovery/backfill operations from completing. Consider adding more capacity."\n summary: "Free space in a pool is too low for recovery/backfill"\n expr: "ceph_health_detail{name=\\"POOL_BACKFILLFULL\\"} > 0"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPoolFull"\n annotations:\n description: "A pool has reached its MAX quota, or OSDs supporting the pool have reached the FULL threshold. Until this is resolved, writes to the pool will be blocked. Pool Breakdown (top 5) {{- range query \\"topk(5, sort_desc(ceph_pool_percent_used * on(pool_id) group_right ceph_pool_metadata))\\" }} - {{ .Labels.name }} at {{ .Value }}% {{- end }} Increase the pool\'s quota, or add capacity to the cluster first then increase the pool\'s quota (e.g. ceph osd pool set quota max_bytes )"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pool-full"\n summary: "Pool is full - writes are blocked"\n expr: "ceph_health_detail{name=\\"POOL_FULL\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.9.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPoolNearFull"\n annotations:\n description: "A pool has exceeded the warning (percent full) threshold, or OSDs supporting the pool have reached the NEARFULL threshold. Writes may continue, but you are at risk of the pool going read-only if more capacity isn\'t made available. Determine the affected pool with \'ceph df detail\', looking at QUOTA BYTES and STORED. Increase the pool\'s quota, or add capacity to the cluster first then increase the pool\'s quota (e.g. ceph osd pool set quota max_bytes ). Also ensure that the balancer is active."\n summary: "One or more Ceph pools are nearly full"\n expr: "ceph_health_detail{name=\\"POOL_NEAR_FULL\\"} > 0"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "healthchecks"\n rules:\n - alert: "CephSlowOps"\n annotations:\n description: "{{ $value }} OSD requests are taking too long to process (osd_op_complaint_time exceeded)"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#slow-ops"\n summary: "OSD operations are slow to complete"\n expr: "ceph_healthcheck_slow_ops > 0"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephDaemonSlowOps"\n annotations:\n description: "{{ $labels.ceph_daemon }} operations are taking too long to process (complaint time exceeded)"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#slow-ops"\n summary: "{{ $labels.ceph_daemon }} operations are slow to complete"\n expr: "ceph_daemon_health_metrics{type=\\"SLOW_OPS\\"} > 0"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "cephadm"\n rules:\n - alert: "CephadmUpgradeFailed"\n annotations:\n description: "The cephadm cluster upgrade process has failed. The cluster remains in an undetermined state. Please review the cephadm logs, to understand the nature of the issue"\n summary: "Ceph version upgrade has failed"\n expr: "ceph_health_detail{name=\\"UPGRADE_EXCEPTION\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.11.2"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephadmDaemonFailed"\n annotations:\n description: "A daemon managed by cephadm is no longer active. Determine, which daemon is down with \'ceph health detail\'. you may start daemons with the \'ceph orch daemon start \'"\n summary: "A ceph daemon managed by cephadm is down"\n expr: "ceph_health_detail{name=\\"CEPHADM_FAILED_DAEMON\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.11.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephadmPaused"\n annotations:\n description: "Cluster management has been paused manually. This will prevent the orchestrator from service management and reconciliation. If this is not intentional, resume cephadm operations with \'ceph orch resume\'"\n documentation: "https://docs.ceph.com/en/latest/cephadm/operations#cephadm-paused"\n summary: "Orchestration tasks via cephadm are PAUSED"\n expr: "ceph_health_detail{name=\\"CEPHADM_PAUSED\\"} > 0"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "hardware"\n rules:\n - alert: "HardwareStorageError"\n annotations:\n description: "Some storage devices are in error. Check `ceph health detail`."\n summary: "Storage devices error(s) detected"\n expr: "ceph_health_detail{name=\\"HARDWARE_STORAGE\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwareMemoryError"\n annotations:\n description: "DIMM error(s) detected. Check `ceph health detail`."\n summary: "DIMM error(s) detected"\n expr: "ceph_health_detail{name=\\"HARDWARE_MEMORY\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.2"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwareProcessorError"\n annotations:\n description: "Processor error(s) detected. Check `ceph health detail`."\n summary: "Processor error(s) detected"\n expr: "ceph_health_detail{name=\\"HARDWARE_PROCESSOR\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.3"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwareNetworkError"\n annotations:\n description: "Network error(s) detected. Check `ceph health detail`."\n summary: "Network error(s) detected"\n expr: "ceph_health_detail{name=\\"HARDWARE_NETWORK\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.4"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwarePowerError"\n annotations:\n description: "Power supply error(s) detected. Check `ceph health detail`."\n summary: "Power supply error(s) detected"\n expr: "ceph_health_detail{name=\\"HARDWARE_POWER\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.5"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwareFanError"\n annotations:\n description: "Fan error(s) detected. Check `ceph health detail`."\n summary: "Fan error(s) detected"\n expr: "ceph_health_detail{name=\\"HARDWARE_FANS\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.6"\n severity: "critical"\n type: "ceph_default"\n - name: "PrometheusServer"\n rules:\n - alert: "PrometheusJobMissing"\n annotations:\n description: "The prometheus job that scrapes from Ceph is no longer defined, this will effectively mean you\'ll have no metrics or alerts for the cluster. Please review the job definitions in the prometheus.yml file of the prometheus instance."\n summary: "The scrape job for Ceph is missing from Prometheus"\n expr: "absent(up{job=\\"ceph\\"})"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.12.1"\n severity: "critical"\n type: "ceph_default"\n - name: "rados"\n rules:\n - alert: "CephObjectMissing"\n annotations:\n description: "The latest version of a RADOS object can not be found, even though all OSDs are up. I/O requests for this object from clients will block (hang). Resolving this issue may require the object to be rolled back to a prior version manually, and manually verified."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#object-unfound"\n summary: "Object(s) marked UNFOUND"\n expr: "(ceph_health_detail{name=\\"OBJECT_UNFOUND\\"} == 1) * on() (count(ceph_osd_up == 1) == bool count(ceph_osd_metadata)) == 1"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.1"\n severity: "critical"\n type: "ceph_default"\n - name: "generic"\n rules:\n - alert: "CephDaemonCrash"\n annotations:\n description: "One or more daemons have crashed recently, and need to be acknowledged. This notification ensures that software crashes do not go unseen. To acknowledge a crash, use the \'ceph crash archive \' command."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks/#recent-crash"\n summary: "One or more Ceph daemons have crashed, and are pending acknowledgement"\n expr: "ceph_health_detail{name=\\"RECENT_CRASH\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.1.2"\n severity: "critical"\n type: "ceph_default"\n - name: "rbdmirror"\n rules:\n - alert: "CephRBDMirrorImagesPerDaemonHigh"\n annotations:\n description: "Number of image replications per daemon is not suppossed to go beyond threshold 100"\n summary: "Number of image replications are now above 100"\n expr: "sum by (ceph_daemon, namespace) (ceph_rbd_mirror_snapshot_image_snapshots) > 100"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.2"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephRBDMirrorImagesNotInSync"\n annotations:\n description: "Both local and remote RBD mirror images should be in sync."\n summary: "Some of the RBD mirror images are not in sync with the remote counter parts."\n expr: "sum by (ceph_daemon, image, namespace, pool) (topk by (ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_local_timestamp) - topk by (ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_remote_timestamp)) != 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.3"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephRBDMirrorImagesNotInSyncVeryHigh"\n annotations:\n description: "More than 10% of the images have synchronization problems"\n summary: "Number of unsynchronized images are very high."\n expr: "count by (ceph_daemon) ((topk by (ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_local_timestamp) - topk by (ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_remote_timestamp)) != 0) > (sum by (ceph_daemon) (ceph_rbd_mirror_snapshot_snapshots)*.1)"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.4"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephRBDMirrorImageTransferBandwidthHigh"\n annotations:\n description: "Detected a heavy increase in bandwidth for rbd replications (over 80%) in the last 30 min. This might not be a problem, but it is good to review the number of images being replicated simultaneously"\n summary: "The replication network usage has been increased over 80% in the last 30 minutes. Review the number of images being replicated. This alert will be cleaned automatically after 30 minutes"\n expr: "rate(ceph_rbd_mirror_journal_replay_bytes[30m]) > 0.80"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.5"\n severity: "warning"\n type: "ceph_default"\n - name: "nvmeof"\n rules:\n - alert: "NVMeoFSubsystemNamespaceLimit"\n annotations:\n description: "Subsystems have a max namespace limit defined at creation time. This alert means that no more namespaces can be added to {{ $labels.nqn }}"\n summary: "{{ $labels.nqn }} subsystem has reached its maximum number of namespaces "\n expr: "(count by(nqn) (ceph_nvmeof_subsystem_namespace_metadata)) >= ceph_nvmeof_subsystem_namespace_limit"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFTooManyGateways"\n annotations:\n description: "You may create many gateways, but 4 is the tested limit"\n summary: "Max supported gateways exceeded "\n expr: "count(ceph_nvmeof_gateway_info) > 4.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFMaxGatewayGroupSize"\n annotations:\n description: "You may create many gateways in a gateway group, but 2 is the tested limit"\n summary: "Max gateways within a gateway group ({{ $labels.group }}) exceeded "\n expr: "count by(group) (ceph_nvmeof_gateway_info) > 2.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFSingleGatewayGroup"\n annotations:\n description: "Although a single member gateway group is valid, it should only be used for test purposes"\n summary: "The gateway group {{ $labels.group }} consists of a single gateway - HA is not possible "\n expr: "count by(group) (ceph_nvmeof_gateway_info) == 1"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighGatewayCPU"\n annotations:\n description: "Typically, high CPU may indicate degraded performance. Consider increasing the number of reactor cores"\n summary: "CPU used by {{ $labels.instance }} NVMe-oF Gateway is high "\n expr: "label_replace(avg by(instance) (rate(ceph_nvmeof_reactor_seconds_total{mode=\\"busy\\"}[1m])),\\"instance\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\") > 80.00"\n for: "10m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFGatewayOpenSecurity"\n annotations:\n description: "It is good practice to ensure subsystems use host security to reduce the risk of unexpected data loss"\n summary: "Subsystem {{ $labels.nqn }} has been defined without host level security "\n expr: "ceph_nvmeof_subsystem_metadata{allow_any_host=\\"yes\\"}"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFTooManySubsystems"\n annotations:\n description: "Although you may continue to create subsystems in {{ $labels.gateway_host }}, the configuration may not be supported"\n summary: "The number of subsystems defined to the gateway exceeds supported values "\n expr: "count by(gateway_host) (label_replace(ceph_nvmeof_subsystem_metadata,\\"gateway_host\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\")) > 16.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFVersionMismatch"\n annotations:\n description: "This may indicate an issue with deployment. Check cephadm logs"\n summary: "The cluster has different NVMe-oF gateway releases active "\n expr: "count(count by(version) (ceph_nvmeof_gateway_info)) > 1"\n for: "1h"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighClientCount"\n annotations:\n description: "The supported limit for clients connecting to a subsystem is 32"\n summary: "The number of clients connected to {{ $labels.nqn }} is too high "\n expr: "ceph_nvmeof_subsystem_host_count > 32.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighHostCPU"\n annotations:\n description: "High CPU on a gateway host can lead to CPU contention and performance degradation"\n summary: "The CPU is high ({{ $value }}%) on NVMeoF Gateway host ({{ $labels.host }}) "\n expr: "100-((100*(avg by(host) (label_replace(rate(node_cpu_seconds_total{mode=\\"idle\\"}[5m]),\\"host\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\")) * on(host) group_right label_replace(ceph_nvmeof_gateway_info,\\"host\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\")))) >= 80.00"\n for: "10m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFInterfaceDown"\n annotations:\n description: "A NIC used by one or more subsystems is in a down state"\n summary: "Network interface {{ $labels.device }} is down "\n expr: "ceph_nvmeof_subsystem_listener_iface_info{operstate=\\"down\\"}"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.14.1"\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFInterfaceDuplex"\n annotations:\n description: "Until this is resolved, performance from the gateway will be degraded"\n summary: "Network interface {{ $labels.device }} is not running in full duplex mode "\n expr: "ceph_nvmeof_subsystem_listener_iface_info{duplex!=\\"full\\"}"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighReadLatency"\n annotations:\n description: "High latencies may indicate a constraint within the cluster e.g. CPU, network. Please investigate"\n summary: "The average read latency over the last 5 mins has reached 10 ms or more on {{ $labels.gateway }}"\n expr: "label_replace((avg by(instance) ((rate(ceph_nvmeof_bdev_read_seconds_total[1m]) / rate(ceph_nvmeof_bdev_reads_completed_total[1m])))),\\"gateway\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\") > 0.01"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighWriteLatency"\n annotations:\n description: "High latencies may indicate a constraint within the cluster e.g. CPU, network. Please investigate"\n summary: "The average write latency over the last 5 mins has reached 20 ms or more on {{ $labels.gateway }}"\n expr: "label_replace((avg by(instance) ((rate(ceph_nvmeof_bdev_write_seconds_total[5m]) / rate(ceph_nvmeof_bdev_writes_completed_total[5m])))),\\"gateway\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\") > 0.02"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n', '/etc/prometheus/alerting/custom_alerts.yml': ''}, 'retention_time': '15d', 'retention_size': '0', 'ip_to_bind_to': ''}} 2026-02-14 01:34:32,150 7f6a18a0de00 DEBUG Determined image: 'quay.io/prometheus/prometheus:v2.51.0' 2026-02-14 01:34:32,164 7f6a18a0de00 INFO Reconfig daemon prometheus.trial014 ... 2026-02-14 01:34:32,362 7f6a18a0de00 DEBUG firewalld.service is not enabled 2026-02-14 01:34:32,362 7f6a18a0de00 DEBUG Not possible to open ports <[9095]>. firewalld.service is not available 2026-02-14 01:34:32,980 7efdc1423e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'ceph-volume', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'lvm', 'zap', '/dev/vg_nvme/lv_4'] 2026-02-14 01:34:33,041 7efdc1423e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:33,041 7efdc1423e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:33,798 7fb874637e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ls'] 2026-02-14 01:34:34,053 7f95c4477e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'orch', 'daemon', 'add', 'osd', 'trial014:vg_nvme/lv_4', '--skip-validation'] 2026-02-14 01:34:34,115 7f95c4477e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:34,115 7f95c4477e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:34,115 7f95c4477e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:34,116 7f95c4477e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid orch daemon add osd trial014:vg_nvme/lv_4 --skip-validation 2026-02-14 01:34:34,603 7f1fcc8bde00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'orch', 'daemon', 'add', 'osd', 'trial014:vg_nvme/lv_4'] 2026-02-14 01:34:34,660 7f1fcc8bde00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:34,660 7f1fcc8bde00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:34,660 7f1fcc8bde00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:34,661 7f1fcc8bde00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid orch daemon add osd trial014:vg_nvme/lv_4 2026-02-14 01:34:34,968 7f90015e7e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--env', 'CEPH_VOLUME_OSDSPEC_AFFINITY=None', '--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--config-json', '-', '--', 'lvm', 'batch', '--no-auto', 'vg_nvme/lv_4', '--yes', '--no-systemd'] 2026-02-14 01:34:35,028 7f90015e7e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:35,029 7f90015e7e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:40,761 7f0d06e7fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'lvm', 'list', '--format', 'json'] 2026-02-14 01:34:40,819 7f0d06e7fe00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:40,819 7f0d06e7fe00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:41,271 7f510afcae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:34:41,287 7f510afcae00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'osd.0', 'image': '', 'deploy_arguments': [], 'params': {'osd_fsid': '1d35940b-bfca-4f46-8c96-9c8c06e3c945', 'allow_ptrace': True}, 'meta': {'service_name': 'osd', 'ports': [], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[osd.0]\n\tkey = AQAr0Y9pqEb8FxAAnMomsBWtNxWw7AGx89cXug==\n\tcaps mgr = "allow profile osd"\n\tcaps mon = "allow profile osd"\n\tcaps osd = "allow *"\n'}} 2026-02-14 01:34:41,287 7f510afcae00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4' 2026-02-14 01:34:41,319 7f510afcae00 INFO Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-osd-0 2026-02-14 01:34:41,319 7f510afcae00 INFO /usr/bin/podman: stderr Error: no such container "ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-osd-0" 2026-02-14 01:34:41,335 7f510afcae00 INFO Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-osd.0 2026-02-14 01:34:41,335 7f510afcae00 INFO /usr/bin/podman: stderr Error: no such container "ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-osd.0" 2026-02-14 01:34:41,335 7f510afcae00 INFO Deploy daemon osd.0 ... 2026-02-14 01:34:41,681 7f510afcae00 DEBUG sysctl: stdout 1048576 2026-02-14 01:34:41,683 7f510afcae00 DEBUG sysctl: stdout 4194304 2026-02-14 01:34:41,831 7f510afcae00 DEBUG Non-zero exit code 1 from systemctl reset-failed ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@osd.0 2026-02-14 01:34:41,832 7f510afcae00 DEBUG systemctl: stderr Failed to reset failed state of unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@osd.0.service: Unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@osd.0.service not loaded. 2026-02-14 01:34:41,958 7f510afcae00 DEBUG systemctl: stderr Created symlink /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460.target.wants/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@osd.0.service → /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@.service. 2026-02-14 01:34:43,235 7f510afcae00 DEBUG firewalld.service is not enabled 2026-02-14 01:34:43,235 7f510afcae00 DEBUG Not possible to enable service . firewalld.service is not available 2026-02-14 01:34:43,365 7f9cda628e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ls'] 2026-02-14 01:34:43,365 7f44f9bd4e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'raw', 'list', '--format', 'json'] 2026-02-14 01:34:43,432 7f44f9bd4e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:43,432 7f44f9bd4e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:44,390 7f931c674e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ls'] 2026-02-14 01:34:44,642 7fe8f5258e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'ceph-volume', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'lvm', 'zap', '/dev/vg_nvme/lv_3'] 2026-02-14 01:34:44,705 7fe8f5258e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:44,705 7fe8f5258e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:45,277 7f2e44a3de00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 01:34:45,335 7f2e44a3de00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:45,335 7f2e44a3de00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:45,741 7ffa4f86fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'orch', 'daemon', 'add', 'osd', 'trial014:vg_nvme/lv_3'] 2026-02-14 01:34:45,801 7ffa4f86fe00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:45,802 7ffa4f86fe00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:45,802 7ffa4f86fe00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:45,802 7ffa4f86fe00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid orch daemon add osd trial014:vg_nvme/lv_3 2026-02-14 01:34:46,116 7f0faa2bee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--env', 'CEPH_VOLUME_OSDSPEC_AFFINITY=None', '--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--config-json', '-', '--', 'lvm', 'batch', '--no-auto', 'vg_nvme/lv_3', '--yes', '--no-systemd'] 2026-02-14 01:34:46,174 7f0faa2bee00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:46,174 7f0faa2bee00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:51,911 7f92a5419e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'lvm', 'list', '--format', 'json'] 2026-02-14 01:34:51,969 7f92a5419e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:51,969 7f92a5419e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:52,427 7f02347dce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:34:52,444 7f02347dce00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'osd.1', 'image': '', 'deploy_arguments': [], 'params': {'osd_fsid': '89abfd8b-9842-4712-b460-553af014b17c', 'allow_ptrace': True}, 'meta': {'service_name': 'osd', 'ports': [], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[osd.1]\n\tkey = AQA20Y9pUXGvIBAARoK0cvUPLYkYVRtXxjUDZw==\n\tcaps mgr = "allow profile osd"\n\tcaps mon = "allow profile osd"\n\tcaps osd = "allow *"\n'}} 2026-02-14 01:34:52,444 7f02347dce00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4' 2026-02-14 01:34:52,475 7f02347dce00 INFO Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-osd-1 2026-02-14 01:34:52,476 7f02347dce00 INFO /usr/bin/podman: stderr Error: no such container "ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-osd-1" 2026-02-14 01:34:52,490 7f02347dce00 INFO Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-osd.1 2026-02-14 01:34:52,490 7f02347dce00 INFO /usr/bin/podman: stderr Error: no such container "ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-osd.1" 2026-02-14 01:34:52,490 7f02347dce00 INFO Deploy daemon osd.1 ... 2026-02-14 01:34:52,819 7f02347dce00 DEBUG sysctl: stdout 1048576 2026-02-14 01:34:52,821 7f02347dce00 DEBUG sysctl: stdout 4194304 2026-02-14 01:34:52,972 7f02347dce00 DEBUG Non-zero exit code 1 from systemctl reset-failed ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@osd.1 2026-02-14 01:34:52,972 7f02347dce00 DEBUG systemctl: stderr Failed to reset failed state of unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@osd.1.service: Unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@osd.1.service not loaded. 2026-02-14 01:34:53,101 7f02347dce00 DEBUG systemctl: stderr Created symlink /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460.target.wants/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@osd.1.service → /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@.service. 2026-02-14 01:34:54,291 7f02347dce00 DEBUG firewalld.service is not enabled 2026-02-14 01:34:54,291 7f02347dce00 DEBUG Not possible to enable service . firewalld.service is not available 2026-02-14 01:34:54,421 7f59a17bbe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ls'] 2026-02-14 01:34:54,421 7ff23ee6ee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'raw', 'list', '--format', 'json'] 2026-02-14 01:34:54,490 7ff23ee6ee00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:54,490 7ff23ee6ee00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:55,394 7fa43f04fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 01:34:55,453 7fa43f04fe00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:55,453 7fa43f04fe00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:55,608 7f122268ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'ceph-volume', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'lvm', 'zap', '/dev/vg_nvme/lv_2'] 2026-02-14 01:34:55,673 7f122268ce00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:55,673 7f122268ce00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:56,241 7f68e0c21e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ls'] 2026-02-14 01:34:57,033 7fd31a5eee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'orch', 'daemon', 'add', 'osd', 'trial014:vg_nvme/lv_2'] 2026-02-14 01:34:57,094 7fd31a5eee00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:57,094 7fd31a5eee00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:34:57,094 7fd31a5eee00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:57,095 7fd31a5eee00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid orch daemon add osd trial014:vg_nvme/lv_2 2026-02-14 01:34:57,405 7f7f100c9e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--env', 'CEPH_VOLUME_OSDSPEC_AFFINITY=None', '--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--config-json', '-', '--', 'lvm', 'batch', '--no-auto', 'vg_nvme/lv_2', '--yes', '--no-systemd'] 2026-02-14 01:34:57,464 7f7f100c9e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:34:57,464 7f7f100c9e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:02,990 7f524a5eee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'lvm', 'list', '--format', 'json'] 2026-02-14 01:35:03,048 7f524a5eee00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:03,048 7f524a5eee00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:03,513 7fc8d2c7be00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:35:03,529 7fc8d2c7be00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'osd.2', 'image': '', 'deploy_arguments': [], 'params': {'osd_fsid': 'ce7d3155-0e14-4989-baf6-12e50590fb75', 'allow_ptrace': True}, 'meta': {'service_name': 'osd', 'ports': [], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[osd.2]\n\tkey = AQBB0Y9pRK6AMhAA3PlNX4pWXFXkKXOn5xJkaA==\n\tcaps mgr = "allow profile osd"\n\tcaps mon = "allow profile osd"\n\tcaps osd = "allow *"\n'}} 2026-02-14 01:35:03,529 7fc8d2c7be00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4' 2026-02-14 01:35:03,559 7fc8d2c7be00 INFO Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-osd-2 2026-02-14 01:35:03,559 7fc8d2c7be00 INFO /usr/bin/podman: stderr Error: no such container "ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-osd-2" 2026-02-14 01:35:03,574 7fc8d2c7be00 INFO Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-osd.2 2026-02-14 01:35:03,574 7fc8d2c7be00 INFO /usr/bin/podman: stderr Error: no such container "ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-osd.2" 2026-02-14 01:35:03,574 7fc8d2c7be00 INFO Deploy daemon osd.2 ... 2026-02-14 01:35:03,903 7fc8d2c7be00 DEBUG sysctl: stdout 1048576 2026-02-14 01:35:03,905 7fc8d2c7be00 DEBUG sysctl: stdout 4194304 2026-02-14 01:35:04,058 7fc8d2c7be00 DEBUG Non-zero exit code 1 from systemctl reset-failed ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@osd.2 2026-02-14 01:35:04,058 7fc8d2c7be00 DEBUG systemctl: stderr Failed to reset failed state of unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@osd.2.service: Unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@osd.2.service not loaded. 2026-02-14 01:35:04,184 7fc8d2c7be00 DEBUG systemctl: stderr Created symlink /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460.target.wants/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@osd.2.service → /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@.service. 2026-02-14 01:35:05,362 7fc8d2c7be00 DEBUG firewalld.service is not enabled 2026-02-14 01:35:05,362 7fc8d2c7be00 DEBUG Not possible to enable service . firewalld.service is not available 2026-02-14 01:35:05,489 7f3b59df8e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ls'] 2026-02-14 01:35:05,490 7f6d90a58e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'raw', 'list', '--format', 'json'] 2026-02-14 01:35:05,555 7f6d90a58e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:05,555 7f6d90a58e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:06,481 7f10e4099e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 01:35:06,539 7f10e4099e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:06,540 7f10e4099e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:27,500 7f23cfdc4e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:35:39,647 7f6b07fbbe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'osd', 'stat', '-f', 'json'] 2026-02-14 01:35:39,710 7f6b07fbbe00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:39,710 7f6b07fbbe00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:39,710 7f6b07fbbe00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:39,711 7f6b07fbbe00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid osd stat -f json 2026-02-14 01:35:41,204 7f576561ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'osd', 'stat', '-f', 'json'] 2026-02-14 01:35:41,264 7f576561ce00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:41,264 7f576561ce00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:41,264 7f576561ce00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:41,265 7f576561ce00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid osd stat -f json 2026-02-14 01:35:42,735 7fa808a1ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'osd', 'stat', '-f', 'json'] 2026-02-14 01:35:42,794 7fa808a1ce00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:42,794 7fa808a1ce00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:42,794 7fa808a1ce00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:42,795 7fa808a1ce00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid osd stat -f json 2026-02-14 01:35:44,258 7f81f3a45e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'osd', 'stat', '-f', 'json'] 2026-02-14 01:35:44,319 7f81f3a45e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:44,319 7f81f3a45e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:44,319 7f81f3a45e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:44,319 7f81f3a45e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid osd stat -f json 2026-02-14 01:35:45,778 7f9082e50e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'osd', 'stat', '-f', 'json'] 2026-02-14 01:35:45,838 7f9082e50e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:45,838 7f9082e50e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:45,838 7f9082e50e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:45,839 7f9082e50e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid osd stat -f json 2026-02-14 01:35:47,301 7fa09142be00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'osd', 'stat', '-f', 'json'] 2026-02-14 01:35:47,359 7fa09142be00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:47,359 7fa09142be00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:47,359 7fa09142be00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:47,359 7fa09142be00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid osd stat -f json 2026-02-14 01:35:47,829 7f4a6de91e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'osd', 'dump', '--format=json'] 2026-02-14 01:35:47,888 7f4a6de91e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:47,888 7f4a6de91e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:47,888 7f4a6de91e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:47,888 7f4a6de91e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid osd dump --format=json 2026-02-14 01:35:48,343 7f4211a91e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'osd', 'pool', 'get', '.mgr', 'pg_num'] 2026-02-14 01:35:48,403 7f4211a91e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:48,403 7f4211a91e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:48,403 7f4211a91e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:48,403 7f4211a91e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid osd pool get .mgr pg_num 2026-02-14 01:35:48,865 7fbd78679e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'auth', 'get-or-create', 'client.0', 'mon', 'allow *', 'osd', 'allow *', 'mds', 'allow *', 'mgr', 'allow *'] 2026-02-14 01:35:48,925 7fbd78679e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:48,925 7fbd78679e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:48,925 7fbd78679e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:48,925 7fbd78679e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid auth get-or-create client.0 mon allow * osd allow * mds allow * mgr allow * 2026-02-14 01:35:50,058 7fbbf37ece00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'config', 'log', '1', '--format=json'] 2026-02-14 01:35:50,116 7fbbf37ece00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:50,116 7fbbf37ece00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:50,116 7fbbf37ece00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:50,117 7fbbf37ece00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid config log 1 --format=json 2026-02-14 01:35:50,572 7f4b1d7dfe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'mgr', 'dump', '--format=json'] 2026-02-14 01:35:50,631 7f4b1d7dfe00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:50,631 7f4b1d7dfe00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:50,631 7f4b1d7dfe00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:50,631 7f4b1d7dfe00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid mgr dump --format=json 2026-02-14 01:35:51,120 7fcb53c02e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'osd', 'dump', '--format=json'] 2026-02-14 01:35:51,180 7fcb53c02e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:51,180 7fcb53c02e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:51,180 7fcb53c02e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:51,181 7fcb53c02e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid osd dump --format=json 2026-02-14 01:35:51,640 7f67c0ccae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'osd', 'dump', '--format=json'] 2026-02-14 01:35:51,702 7f67c0ccae00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:51,702 7f67c0ccae00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:51,702 7f67c0ccae00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:51,702 7f67c0ccae00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid osd dump --format=json 2026-02-14 01:35:52,179 7f6e178cfe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'tell', 'osd.0', 'flush_pg_stats'] 2026-02-14 01:35:52,184 7f588ba51e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'tell', 'osd.1', 'flush_pg_stats'] 2026-02-14 01:35:52,185 7faf8d9f2e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'tell', 'osd.5', 'flush_pg_stats'] 2026-02-14 01:35:52,185 7f7ab0cd3e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'tell', 'osd.4', 'flush_pg_stats'] 2026-02-14 01:35:52,189 7f046f1f4e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'tell', 'osd.2', 'flush_pg_stats'] 2026-02-14 01:35:52,189 7f17345fae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'tell', 'osd.3', 'flush_pg_stats'] 2026-02-14 01:35:52,257 7f6e178cfe00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:52,257 7f6e178cfe00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:52,257 7f6e178cfe00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:52,257 7f6e178cfe00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid tell osd.0 flush_pg_stats 2026-02-14 01:35:52,264 7f17345fae00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:52,264 7f17345fae00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:52,264 7f17345fae00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:52,264 7f046f1f4e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:52,264 7f046f1f4e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:52,264 7f046f1f4e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:52,264 7faf8d9f2e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:52,264 7f046f1f4e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid tell osd.2 flush_pg_stats 2026-02-14 01:35:52,264 7faf8d9f2e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:52,265 7faf8d9f2e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:52,265 7f588ba51e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:52,265 7f588ba51e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:52,265 7f588ba51e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:52,265 7faf8d9f2e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid tell osd.5 flush_pg_stats 2026-02-14 01:35:52,265 7f588ba51e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid tell osd.1 flush_pg_stats 2026-02-14 01:35:52,265 7f7ab0cd3e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:52,266 7f7ab0cd3e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:52,266 7f7ab0cd3e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:52,266 7f7ab0cd3e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid tell osd.4 flush_pg_stats 2026-02-14 01:35:52,267 7f17345fae00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid tell osd.3 flush_pg_stats 2026-02-14 01:35:52,692 7ff7a1c7fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'osd', 'last-stat-seq', 'osd.1'] 2026-02-14 01:35:52,713 7f675f08ee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'osd', 'last-stat-seq', 'osd.2'] 2026-02-14 01:35:52,716 7fe48e48ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'osd', 'last-stat-seq', 'osd.4'] 2026-02-14 01:35:52,717 7f2eb61fae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'osd', 'last-stat-seq', 'osd.3'] 2026-02-14 01:35:52,718 7fdf77690e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'osd', 'last-stat-seq', 'osd.0'] 2026-02-14 01:35:52,769 7f6a08c06e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'osd', 'last-stat-seq', 'osd.5'] 2026-02-14 01:35:52,776 7ff7a1c7fe00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:52,776 7ff7a1c7fe00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:52,776 7ff7a1c7fe00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:52,776 7ff7a1c7fe00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid osd last-stat-seq osd.1 2026-02-14 01:35:52,818 7fdf77690e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:52,819 7fdf77690e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:52,819 7fdf77690e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:52,819 7fdf77690e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid osd last-stat-seq osd.0 2026-02-14 01:35:52,819 7fe48e48ae00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:52,820 7fe48e48ae00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:52,820 7fe48e48ae00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:52,820 7fe48e48ae00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid osd last-stat-seq osd.4 2026-02-14 01:35:52,822 7f2eb61fae00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:52,822 7f2eb61fae00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:52,822 7f2eb61fae00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:52,823 7f2eb61fae00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid osd last-stat-seq osd.3 2026-02-14 01:35:52,857 7f675f08ee00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:52,858 7f675f08ee00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:52,858 7f675f08ee00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:52,858 7f675f08ee00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid osd last-stat-seq osd.2 2026-02-14 01:35:52,890 7f6a08c06e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:52,890 7f6a08c06e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:52,890 7f6a08c06e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:52,891 7f6a08c06e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid osd last-stat-seq osd.5 2026-02-14 01:35:54,284 7f2c3546ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'osd', 'last-stat-seq', 'osd.1'] 2026-02-14 01:35:54,324 7fb4e15c0e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'osd', 'last-stat-seq', 'osd.3'] 2026-02-14 01:35:54,328 7fa197eb3e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'osd', 'last-stat-seq', 'osd.0'] 2026-02-14 01:35:54,328 7fac1f7cde00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'osd', 'last-stat-seq', 'osd.4'] 2026-02-14 01:35:54,359 7f2c3546ae00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:54,359 7f2c3546ae00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:54,359 7f2c3546ae00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:54,359 7f2c3546ae00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid osd last-stat-seq osd.1 2026-02-14 01:35:54,379 7f604e287e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'osd', 'last-stat-seq', 'osd.2'] 2026-02-14 01:35:54,387 7fb4e15c0e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:54,387 7fb4e15c0e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:54,388 7fb4e15c0e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:54,388 7fb4e15c0e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid osd last-stat-seq osd.3 2026-02-14 01:35:54,422 7f170e65fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'osd', 'last-stat-seq', 'osd.5'] 2026-02-14 01:35:54,437 7fac1f7cde00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:54,437 7fac1f7cde00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:54,437 7fac1f7cde00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:54,438 7fac1f7cde00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid osd last-stat-seq osd.4 2026-02-14 01:35:54,438 7fa197eb3e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:54,438 7fa197eb3e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:54,438 7fa197eb3e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:54,439 7fa197eb3e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid osd last-stat-seq osd.0 2026-02-14 01:35:54,464 7f604e287e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:54,464 7f604e287e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:54,464 7f604e287e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:54,465 7f604e287e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid osd last-stat-seq osd.2 2026-02-14 01:35:54,513 7f170e65fe00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:54,514 7f170e65fe00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:54,514 7f170e65fe00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:54,514 7f170e65fe00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid osd last-stat-seq osd.5 2026-02-14 01:35:55,027 7f10ee275e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'pg', 'dump', '--format=json'] 2026-02-14 01:35:55,085 7f10ee275e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:55,085 7f10ee275e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:55,085 7f10ee275e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:55,086 7f10ee275e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid pg dump --format=json 2026-02-14 01:35:55,541 7f0b45c19e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'pg', 'dump', '--format=json'] 2026-02-14 01:35:55,600 7f0b45c19e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:55,600 7f0b45c19e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:55,600 7f0b45c19e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:55,601 7f0b45c19e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid pg dump --format=json 2026-02-14 01:35:56,055 7f3de35f8e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'health', '--format=json'] 2026-02-14 01:35:56,113 7f3de35f8e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:56,113 7f3de35f8e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:56,113 7f3de35f8e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:56,113 7f3de35f8e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid health --format=json 2026-02-14 01:35:56,619 7fc35fc9fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'bash', '-c', 'ceph config set mgr mgr/cephadm/use_repo_digest true --force'] 2026-02-14 01:35:56,676 7fc35fc9fe00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:56,676 7fc35fc9fe00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:56,677 7fc35fc9fe00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:56,677 7fc35fc9fe00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint bash quay.ceph.io/ceph-ci/ceph:squid -c ceph config set mgr mgr/cephadm/use_repo_digest true --force 2026-02-14 01:35:57,183 7f4a719dae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'bash', '-c', 'ceph orch status'] 2026-02-14 01:35:57,242 7f4a719dae00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:57,243 7f4a719dae00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:57,243 7f4a719dae00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:57,243 7f4a719dae00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint bash quay.ceph.io/ceph-ci/ceph:squid -c ceph orch status 2026-02-14 01:35:57,711 7ff8029c1e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'bash', '-c', 'ceph orch ps'] 2026-02-14 01:35:57,770 7ff8029c1e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:57,770 7ff8029c1e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:57,770 7ff8029c1e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:57,771 7ff8029c1e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint bash quay.ceph.io/ceph-ci/ceph:squid -c ceph orch ps 2026-02-14 01:35:58,238 7fe34380be00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'bash', '-c', 'ceph orch ls'] 2026-02-14 01:35:58,297 7fe34380be00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:58,297 7fe34380be00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:58,297 7fe34380be00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:58,298 7fe34380be00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint bash quay.ceph.io/ceph-ci/ceph:squid -c ceph orch ls 2026-02-14 01:35:58,762 7f2ca3e5de00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'bash', '-c', 'ceph orch host ls'] 2026-02-14 01:35:58,821 7f2ca3e5de00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:58,821 7f2ca3e5de00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:58,821 7f2ca3e5de00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:58,822 7f2ca3e5de00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint bash quay.ceph.io/ceph-ci/ceph:squid -c ceph orch host ls 2026-02-14 01:35:59,290 7f536a3ede00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'bash', '-c', 'ceph orch device ls'] 2026-02-14 01:35:59,349 7f536a3ede00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:59,349 7f536a3ede00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:59,349 7f536a3ede00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:59,349 7f536a3ede00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint bash quay.ceph.io/ceph-ci/ceph:squid -c ceph orch device ls 2026-02-14 01:35:59,839 7fc3e79ebe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'bash', '-c', 'ceph fs volume create cephfs --placement=4'] 2026-02-14 01:35:59,898 7fc3e79ebe00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:59,898 7fc3e79ebe00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:35:59,898 7fc3e79ebe00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:35:59,899 7fc3e79ebe00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint bash quay.ceph.io/ceph-ci/ceph:squid -c ceph fs volume create cephfs --placement=4 2026-02-14 01:36:01,520 7fd82641ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:36:01,536 7fd82641ae00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'mds.cephfs.trial014.ubresr', 'image': '', 'deploy_arguments': [], 'params': {'allow_ptrace': True}, 'meta': {'service_name': 'mds.cephfs', 'ports': [], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[mds.cephfs.trial014.ubresr]\nkey = AQCB0Y9paSGPGBAAe4qWrzsf3JPgerwk3WRkCA==\n'}} 2026-02-14 01:36:01,537 7fd82641ae00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4' 2026-02-14 01:36:01,571 7fd82641ae00 INFO Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-mds-cephfs-trial014-ubresr 2026-02-14 01:36:01,571 7fd82641ae00 INFO /usr/bin/podman: stderr Error: no such container "ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-mds-cephfs-trial014-ubresr" 2026-02-14 01:36:01,586 7fd82641ae00 INFO Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-mds.cephfs.trial014.ubresr 2026-02-14 01:36:01,586 7fd82641ae00 INFO /usr/bin/podman: stderr Error: no such container "ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-mds.cephfs.trial014.ubresr" 2026-02-14 01:36:01,586 7fd82641ae00 INFO Deploy daemon mds.cephfs.trial014.ubresr ... 2026-02-14 01:36:01,726 7f66b7a62e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'bash', '-c', 'ceph fs dump'] 2026-02-14 01:36:01,791 7f66b7a62e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:01,791 7f66b7a62e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:36:01,791 7f66b7a62e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:01,792 7f66b7a62e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint bash quay.ceph.io/ceph-ci/ceph:squid -c ceph fs dump 2026-02-14 01:36:01,863 7fd82641ae00 DEBUG Non-zero exit code 1 from systemctl reset-failed ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@mds.cephfs.trial014.ubresr 2026-02-14 01:36:01,864 7fd82641ae00 DEBUG systemctl: stderr Failed to reset failed state of unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@mds.cephfs.trial014.ubresr.service: Unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@mds.cephfs.trial014.ubresr.service not loaded. 2026-02-14 01:36:01,999 7fd82641ae00 DEBUG systemctl: stderr Created symlink /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460.target.wants/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@mds.cephfs.trial014.ubresr.service → /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@.service. 2026-02-14 01:36:02,163 7fd82641ae00 DEBUG firewalld.service is not enabled 2026-02-14 01:36:02,164 7fd82641ae00 DEBUG Not possible to enable service . firewalld.service is not available 2026-02-14 01:36:02,326 7f71b9697e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'bash', '-c', 'ceph fs set cephfs max_mds 2'] 2026-02-14 01:36:02,386 7f71b9697e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:02,386 7f71b9697e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:36:02,386 7f71b9697e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:02,386 7f71b9697e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint bash quay.ceph.io/ceph-ci/ceph:squid -c ceph fs set cephfs max_mds 2 2026-02-14 01:36:02,967 7fc03de05e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:36:02,984 7fc03de05e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'mds.cephfs.trial014.kwvbap', 'image': '', 'deploy_arguments': [], 'params': {'allow_ptrace': True}, 'meta': {'service_name': 'mds.cephfs', 'ports': [], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[mds.cephfs.trial014.kwvbap]\nkey = AQCC0Y9pGiBGMxAA7p0e98mGukEB03+oiEk3QQ==\n'}} 2026-02-14 01:36:02,984 7fc03de05e00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4' 2026-02-14 01:36:03,015 7fc03de05e00 INFO Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-mds-cephfs-trial014-kwvbap 2026-02-14 01:36:03,015 7fc03de05e00 INFO /usr/bin/podman: stderr Error: no such container "ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-mds-cephfs-trial014-kwvbap" 2026-02-14 01:36:03,029 7fc03de05e00 INFO Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-mds.cephfs.trial014.kwvbap 2026-02-14 01:36:03,030 7fc03de05e00 INFO /usr/bin/podman: stderr Error: no such container "ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-mds.cephfs.trial014.kwvbap" 2026-02-14 01:36:03,030 7fc03de05e00 INFO Deploy daemon mds.cephfs.trial014.kwvbap ... 2026-02-14 01:36:03,267 7fc03de05e00 DEBUG Non-zero exit code 1 from systemctl reset-failed ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@mds.cephfs.trial014.kwvbap 2026-02-14 01:36:03,267 7fc03de05e00 DEBUG systemctl: stderr Failed to reset failed state of unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@mds.cephfs.trial014.kwvbap.service: Unit ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@mds.cephfs.trial014.kwvbap.service not loaded. 2026-02-14 01:36:03,400 7fc03de05e00 DEBUG systemctl: stderr Created symlink /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460.target.wants/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@mds.cephfs.trial014.kwvbap.service → /etc/systemd/system/ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460@.service. 2026-02-14 01:36:03,558 7fc03de05e00 DEBUG firewalld.service is not enabled 2026-02-14 01:36:03,558 7fc03de05e00 DEBUG Not possible to enable service . firewalld.service is not available 2026-02-14 01:36:03,733 7fbe3d054e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'bash', '-c', 'ceph fs set cephfs allow_standby_replay true'] 2026-02-14 01:36:03,796 7fbe3d054e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:03,796 7fbe3d054e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:36:03,796 7fbe3d054e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:03,796 7fbe3d054e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint bash quay.ceph.io/ceph-ci/ceph:squid -c ceph fs set cephfs allow_standby_replay true 2026-02-14 01:36:04,423 7fb4b48d6e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ls'] 2026-02-14 01:36:04,736 7f4395df7e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'bash', '-c', 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it'] 2026-02-14 01:36:04,803 7f4395df7e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:04,803 7f4395df7e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:36:04,803 7f4395df7e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:04,804 7f4395df7e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint bash quay.ceph.io/ceph-ci/ceph:squid -c ceph fs set cephfs inline_data true --yes-i-really-really-mean-it 2026-02-14 01:36:05,476 7f5b3fc31e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ls'] 2026-02-14 01:36:06,254 7f4b98e39e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'bash', '-c', 'ceph fs dump'] 2026-02-14 01:36:06,318 7f4b98e39e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:06,318 7f4b98e39e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:36:06,318 7f4b98e39e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:06,319 7f4b98e39e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint bash quay.ceph.io/ceph-ci/ceph:squid -c ceph fs dump 2026-02-14 01:36:06,805 7ffb0f681e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'bash', '-c', 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"'] 2026-02-14 01:36:06,868 7ffb0f681e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:06,868 7ffb0f681e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:36:06,868 7ffb0f681e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:06,868 7ffb0f681e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint bash quay.ceph.io/ceph-ci/ceph:squid -c ceph --format=json fs dump | jq -e ".filesystems | length == 1" 2026-02-14 01:36:07,343 7fc3a1686e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'bash', '-c', 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done'] 2026-02-14 01:36:07,404 7fc3a1686e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:07,404 7fc3a1686e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:36:07,404 7fc3a1686e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:07,404 7fc3a1686e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint bash quay.ceph.io/ceph-ci/ceph:squid -c while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done 2026-02-14 01:36:07,922 7f54a8027e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'fs', 'dump', '--format=json'] 2026-02-14 01:36:07,984 7f54a8027e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:07,985 7f54a8027e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:36:07,985 7f54a8027e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:07,985 7f54a8027e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid fs dump --format=json 2026-02-14 01:36:08,696 7f165968fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'fs', 'ls'] 2026-02-14 01:36:08,759 7f165968fe00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:08,760 7f165968fe00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:36:08,760 7f165968fe00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:08,760 7f165968fe00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid fs ls 2026-02-14 01:36:11,022 7f526ee04e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'ceph', 'fs', 'ls'] 2026-02-14 01:36:11,087 7f526ee04e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:11,087 7f526ee04e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:36:11,088 7f526ee04e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:11,088 7f526ee04e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:squid fs ls 2026-02-14 01:36:13,362 7f9e4ce6de00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '-e', 'sha1=eae49298735c216a37f3957204b0fe058b58e376', '--', 'bash', '-c', 'ceph config set mgr mgr/orchestrator/fail_fs false || true'] 2026-02-14 01:36:13,426 7f9e4ce6de00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:13,426 7f9e4ce6de00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:36:13,427 7f9e4ce6de00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:13,427 7f9e4ce6de00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -e sha1=eae49298735c216a37f3957204b0fe058b58e376 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint bash quay.ceph.io/ceph-ci/ceph:squid -c ceph config set mgr mgr/orchestrator/fail_fs false || true 2026-02-14 01:36:13,942 7f1964844e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '-e', 'sha1=eae49298735c216a37f3957204b0fe058b58e376', '--', 'bash', '-c', 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force'] 2026-02-14 01:36:14,002 7f1964844e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:14,002 7f1964844e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:36:14,002 7f1964844e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:14,003 7f1964844e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -e sha1=eae49298735c216a37f3957204b0fe058b58e376 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint bash quay.ceph.io/ceph-ci/ceph:squid -c ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force 2026-02-14 01:36:14,509 7efc1a276e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '-e', 'sha1=eae49298735c216a37f3957204b0fe058b58e376', '--', 'bash', '-c', 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force'] 2026-02-14 01:36:14,571 7efc1a276e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:14,571 7efc1a276e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:36:14,571 7efc1a276e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:14,572 7efc1a276e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -e sha1=eae49298735c216a37f3957204b0fe058b58e376 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint bash quay.ceph.io/ceph-ci/ceph:squid -c ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force 2026-02-14 01:36:15,045 7f978ae8ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '-e', 'sha1=eae49298735c216a37f3957204b0fe058b58e376', '--', 'bash', '-c', 'ceph config set global log_to_journald false --force'] 2026-02-14 01:36:15,104 7f978ae8ae00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:15,105 7f978ae8ae00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:36:15,105 7f978ae8ae00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:15,105 7f978ae8ae00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -e sha1=eae49298735c216a37f3957204b0fe058b58e376 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint bash quay.ceph.io/ceph-ci/ceph:squid -c ceph config set global log_to_journald false --force 2026-02-14 01:36:15,572 7f2c3181ee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '-e', 'sha1=eae49298735c216a37f3957204b0fe058b58e376', '--', 'bash', '-c', 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1'] 2026-02-14 01:36:15,633 7f2c3181ee00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:15,633 7f2c3181ee00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:36:15,633 7f2c3181ee00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:15,634 7f2c3181ee00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -e sha1=eae49298735c216a37f3957204b0fe058b58e376 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint bash quay.ceph.io/ceph-ci/ceph:squid -c ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 2026-02-14 01:36:16,013 7f7a5ba14e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:eae49298735c216a37f3957204b0fe058b58e376', '--timeout', '895', 'pull'] 2026-02-14 01:36:16,031 7f7a5ba14e00 INFO Pulling container image quay.ceph.io/ceph-ci/ceph:eae49298735c216a37f3957204b0fe058b58e376... 2026-02-14 01:36:16,115 7f7fcd663e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:squid', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '-e', 'sha1=eae49298735c216a37f3957204b0fe058b58e376', '--', 'bash', '-c', "while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done"] 2026-02-14 01:36:16,175 7f7fcd663e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:16,175 7f7fcd663e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:36:16,175 7f7fcd663e00 DEBUG Using specified config: /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:16,176 7f7fcd663e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:squid -e NODE_NAME=trial014 -e sha1=eae49298735c216a37f3957204b0fe058b58e376 -v /var/run/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint bash quay.ceph.io/ceph-ci/ceph:squid -c while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done 2026-02-14 01:36:26,449 7f7a5ba14e00 DEBUG /usr/bin/podman: stdout edc95c9460f10a90f1e921f9347538ed9be0b1c7aee54201d89be4c7dacb23c7,[quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9] 2026-02-14 01:36:26,595 7f7a5ba14e00 DEBUG ceph: stdout ceph version 20.2.0-667-geae49298 (eae49298735c216a37f3957204b0fe058b58e376) tentacle (stable) 2026-02-14 01:36:36,555 7f7d78fdae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:36:41,839 7fb9077ebe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ls'] 2026-02-14 01:36:42,852 7f09a9228e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:36:43,084 7f281cc75e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'list-networks'] 2026-02-14 01:36:44,227 7fe83ac23e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:36:44,244 7fe83ac23e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'ceph-exporter.trial014', 'image': '', 'deploy_arguments': [], 'params': {'tcp_ports': [9926], 'allow_ptrace': True}, 'meta': {'service_name': 'ceph-exporter', 'ports': [9926], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[client.ceph-exporter.trial014]\nkey = AQAL0Y9pSQnMEBAAd1kOyDHxNKSk3p930enz8A==\n', 'prio-limit': '5', 'stats-period': '5'}} 2026-02-14 01:36:44,244 7fe83ac23e00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4' 2026-02-14 01:36:44,258 7fe83ac23e00 INFO Redeploy daemon ceph-exporter.trial014 ... 2026-02-14 01:36:44,424 7fe83ac23e00 INFO Writing ceph-exporter config... 2026-02-14 01:36:45,089 7fe83ac23e00 DEBUG firewalld.service is not enabled 2026-02-14 01:36:45,089 7fe83ac23e00 DEBUG Not possible to open ports <[9926]>. firewalld.service is not available 2026-02-14 01:36:45,371 7f9d5b64ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.io/prometheus/prometheus:v2.51.0', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:36:45,390 7f9d5b64ce00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'prometheus.trial014', 'image': '', 'deploy_arguments': [], 'params': {'tcp_ports': [9095], 'reconfig': True, 'allow_ptrace': True}, 'meta': {'service_name': 'prometheus', 'ports': [9095], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'files': {'prometheus.yml': "# This file is generated by cephadm.\nglobal:\n scrape_interval: 10s\n evaluation_interval: 10s\n external_labels:\n cluster: 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\nrule_files:\n - /etc/prometheus/alerting/*\n\nalerting:\n alertmanagers:\n - scheme: http\n http_sd_configs:\n - url: http://10.20.193.30:8765/sd/prometheus/sd-config?service=alertmanager\n\nscrape_configs:\n - job_name: 'ceph'\n relabel_configs:\n - source_labels: [__address__]\n target_label: cluster\n replacement: 29ce0a10-0945-11f1-b2be-d404e6e7d460\n - source_labels: [instance]\n target_label: instance\n replacement: 'ceph_cluster'\n honor_labels: true\n http_sd_configs:\n - url: http://10.20.193.30:8765/sd/prometheus/sd-config?service=mgr-prometheus\n\n - job_name: 'node'\n relabel_configs:\n - source_labels: [__address__]\n target_label: cluster\n replacement: 29ce0a10-0945-11f1-b2be-d404e6e7d460\n http_sd_configs:\n - url: http://10.20.193.30:8765/sd/prometheus/sd-config?service=node-exporter\n\n\n - job_name: 'ceph-exporter'\n relabel_configs:\n - source_labels: [__address__]\n target_label: cluster\n replacement: 29ce0a10-0945-11f1-b2be-d404e6e7d460\n honor_labels: true\n http_sd_configs:\n - url: http://10.20.193.30:8765/sd/prometheus/sd-config?service=ceph-exporter\n\n - job_name: 'nvmeof'\n http_sd_configs:\n - url: http://10.20.193.30:8765/sd/prometheus/sd-config?service=nvmeof\n\n - job_name: 'nfs'\n http_sd_configs:\n - url: http://10.20.193.30:8765/sd/prometheus/sd-config?service=nfs\n\n - job_name: 'smb'\n http_sd_configs:\n - url: http://10.20.193.30:8765/sd/prometheus/sd-config?service=smb\n\n", '/etc/prometheus/alerting/ceph_alerts.yml': 'groups:\n - name: "cluster health"\n rules:\n - alert: "CephHealthError"\n annotations:\n description: "The cluster state has been HEALTH_ERROR for more than 5 minutes on cluster {{ $labels.cluster }}. Please check \'ceph health detail\' for more information."\n summary: "Ceph is in the ERROR state on cluster {{ $labels.cluster }}"\n expr: "ceph_health_status == 2"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.2.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephHealthWarning"\n annotations:\n description: "The cluster state has been HEALTH_WARN for more than 15 minutes on cluster {{ $labels.cluster }}. Please check \'ceph health detail\' for more information."\n summary: "Ceph is in the WARNING state on cluster {{ $labels.cluster }}"\n expr: "ceph_health_status == 1"\n for: "15m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "mon"\n rules:\n - alert: "CephMonDownQuorumAtRisk"\n annotations:\n description: "{{ $min := printf \\"floor(count(ceph_mon_metadata{cluster=\'%s\'}) / 2) + 1\\" .Labels.cluster | query | first | value }}Quorum requires a majority of monitors (x {{ $min }}) to be active. Without quorum the cluster will become inoperable, affecting all services and connected clients. The following monitors are down: {{- range printf \\"(ceph_mon_quorum_status{cluster=\'%s\'} == 0) + on(cluster,ceph_daemon) group_left(hostname) (ceph_mon_metadata * 0)\\" .Labels.cluster | query }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-down"\n summary: "Monitor quorum is at risk on cluster {{ $labels.cluster }}"\n expr: |\n (\n (ceph_health_detail{name="MON_DOWN"} == 1) * on() group_right(cluster) (\n count(ceph_mon_quorum_status == 1) by(cluster)== bool (floor(count(ceph_mon_metadata) by(cluster) / 2) + 1)\n )\n ) == 1\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.3.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephMonDown"\n annotations:\n description: "{{ $down := printf \\"count(ceph_mon_quorum_status{cluster=\'%s\'} == 0)\\" .Labels.cluster | query | first | value }}{{ $s := \\"\\" }}{{ if gt $down 1.0 }}{{ $s = \\"s\\" }}{{ end }}You have {{ $down }} monitor{{ $s }} down. Quorum is still intact, but the loss of an additional monitor will make your cluster inoperable. The following monitors are down: {{- range printf \\"(ceph_mon_quorum_status{cluster=\'%s\'} == 0) + on(cluster,ceph_daemon) group_left(hostname) (ceph_mon_metadata * 0)\\" .Labels.cluster | query }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-down"\n summary: "One or more monitors down on cluster {{ $labels.cluster }}"\n expr: |\n (count by (cluster) (ceph_mon_quorum_status == 0)) <= (count by (cluster) (ceph_mon_metadata) - floor((count by (cluster) (ceph_mon_metadata) / 2 + 1)))\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephMonDiskspaceCritical"\n annotations:\n description: "The free space available to a monitor\'s store is critically low. You should increase the space available to the monitor(s). The default directory is /var/lib/ceph/mon-*/data/store.db on traditional deployments, and /var/lib/rook/mon-*/data/store.db on the mon pod\'s worker node for Rook. Look for old, rotated versions of *.log and MANIFEST*. Do NOT touch any *.sst files. Also check any other directories under /var/lib/rook and other directories on the same filesystem, often /var/log and /var/tmp are culprits. Your monitor hosts are; {{- range query \\"ceph_mon_metadata\\"}} - {{ .Labels.hostname }} {{- end }}"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-disk-crit"\n summary: "Filesystem space on at least one monitor is critically low on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"MON_DISK_CRIT\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.3.2"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephMonDiskspaceLow"\n annotations:\n description: "The space available to a monitor\'s store is approaching full (>70% is the default). You should increase the space available to the monitor(s). The default directory is /var/lib/ceph/mon-*/data/store.db on traditional deployments, and /var/lib/rook/mon-*/data/store.db on the mon pod\'s worker node for Rook. Look for old, rotated versions of *.log and MANIFEST*. Do NOT touch any *.sst files. Also check any other directories under /var/lib/rook and other directories on the same filesystem, often /var/log and /var/tmp are culprits. Your monitor hosts are; {{- range query \\"ceph_mon_metadata\\"}} - {{ .Labels.hostname }} {{- end }}"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-disk-low"\n summary: "Drive space on at least one monitor is approaching full on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"MON_DISK_LOW\\"} == 1"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephMonClockSkew"\n annotations:\n description: "Ceph monitors rely on closely synchronized time to maintain quorum and cluster consistency. This event indicates that the time on at least one mon has drifted too far from the lead mon. Review cluster status with ceph -s. This will show which monitors are affected. Check the time sync status on each monitor host with \'ceph time-sync-status\' and the state and peers of your ntpd or chrony daemon."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-clock-skew"\n summary: "Clock skew detected among monitors on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"MON_CLOCK_SKEW\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "osd"\n rules:\n - alert: "CephOSDDownHigh"\n annotations:\n description: "{{ $value | humanize }}% or {{ with printf \\"count (ceph_osd_up{cluster=\'%s\'} == 0)\\" .Labels.cluster | query }}{{ . | first | value }}{{ end }} of {{ with printf \\"count (ceph_osd_up{cluster=\'%s\'})\\" .Labels.cluster | query }}{{ . | first | value }}{{ end }} OSDs are down (>= 10%). The following OSDs are down: {{- range printf \\"(ceph_osd_up{cluster=\'%s\'} * on(cluster, ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0\\" .Labels.cluster | query }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}"\n summary: "More than 10% of OSDs are down on cluster {{ $labels.cluster }}"\n expr: "count by (cluster) (ceph_osd_up == 0) / count by (cluster) (ceph_osd_up) * 100 >= 10"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephOSDHostDown"\n annotations:\n description: "The following OSDs are down: {{- range printf \\"(ceph_osd_up{cluster=\'%s\'} * on(cluster,ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0\\" .Labels.cluster | query }} - {{ .Labels.hostname }} : {{ .Labels.ceph_daemon }} {{- end }}"\n summary: "An OSD host is offline on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"OSD_HOST_DOWN\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.8"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDDown"\n annotations:\n description: "{{ $num := printf \\"count(ceph_osd_up{cluster=\'%s\'} == 0) \\" .Labels.cluster | query | first | value }}{{ $s := \\"\\" }}{{ if gt $num 1.0 }}{{ $s = \\"s\\" }}{{ end }}{{ $num }} OSD{{ $s }} down for over 5mins. The following OSD{{ $s }} {{ if eq $s \\"\\" }}is{{ else }}are{{ end }} down: {{- range printf \\"(ceph_osd_up{cluster=\'%s\'} * on(cluster,ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0\\" .Labels.cluster | query }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-down"\n summary: "An OSD has been marked down on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"OSD_DOWN\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.2"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDNearFull"\n annotations:\n description: "One or more OSDs have reached the NEARFULL threshold. Use \'ceph health detail\' and \'ceph osd df\' to identify the problem. To resolve, add capacity to the affected OSD\'s failure domain, restore down/out OSDs, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-nearfull"\n summary: "OSD(s) running low on free space (NEARFULL) on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"OSD_NEARFULL\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.3"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDFull"\n annotations:\n description: "An OSD has reached the FULL threshold. Writes to pools that share the affected OSD will be blocked. Use \'ceph health detail\' and \'ceph osd df\' to identify the problem. To resolve, add capacity to the affected OSD\'s failure domain, restore down/out OSDs, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-full"\n summary: "OSD full, writes blocked on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"OSD_FULL\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.6"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephOSDBackfillFull"\n annotations:\n description: "An OSD has reached the BACKFILL FULL threshold. This will prevent rebalance operations from completing. Use \'ceph health detail\' and \'ceph osd df\' to identify the problem. To resolve, add capacity to the affected OSD\'s failure domain, restore down/out OSDs, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-backfillfull"\n summary: "OSD(s) too full for backfill operations on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"OSD_BACKFILLFULL\\"} > 0"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDTooManyRepairs"\n annotations:\n description: "Reads from an OSD have used a secondary PG to return data to the client, indicating a potential failing drive."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-too-many-repairs"\n summary: "OSD reports a high number of read errors on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"OSD_TOO_MANY_REPAIRS\\"} == 1"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDTimeoutsPublicNetwork"\n annotations:\n description: "OSD heartbeats on the cluster\'s \'public\' network (frontend) are running slow. Investigate the network for latency or loss issues. Use \'ceph health detail\' to show the affected OSDs."\n summary: "Network issues delaying OSD heartbeats (public network) on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"OSD_SLOW_PING_TIME_FRONT\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDTimeoutsClusterNetwork"\n annotations:\n description: "OSD heartbeats on the cluster\'s \'cluster\' network (backend) are slow. Investigate the network for latency issues on this subnet. Use \'ceph health detail\' to show the affected OSDs."\n summary: "Network issues delaying OSD heartbeats (cluster network) on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"OSD_SLOW_PING_TIME_BACK\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDInternalDiskSizeMismatch"\n annotations:\n description: "One or more OSDs have an internal inconsistency between metadata and the size of the device. This could lead to the OSD(s) crashing in future. You should redeploy the affected OSDs."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#bluestore-disk-size-mismatch"\n summary: "OSD size inconsistency error on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"BLUESTORE_DISK_SIZE_MISMATCH\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephDeviceFailurePredicted"\n annotations:\n description: "The device health module has determined that one or more devices will fail soon. To review device status use \'ceph device ls\'. To show a specific device use \'ceph device info \'. Mark the OSD out so that data may migrate to other OSDs. Once the OSD has drained, destroy the OSD, replace the device, and redeploy the OSD."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#id2"\n summary: "Device(s) predicted to fail soon on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"DEVICE_HEALTH\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephDeviceFailurePredictionTooHigh"\n annotations:\n description: "The device health module has determined that devices predicted to fail can not be remediated automatically, since too many OSDs would be removed from the cluster to ensure performance and availability. Prevent data integrity issues by adding new OSDs so that data may be relocated."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#device-health-toomany"\n summary: "Too many devices are predicted to fail on cluster {{ $labels.cluster }}, unable to resolve"\n expr: "ceph_health_detail{name=\\"DEVICE_HEALTH_TOOMANY\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.7"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephDeviceFailureRelocationIncomplete"\n annotations:\n description: "The device health module has determined that one or more devices will fail soon, but the normal process of relocating the data on the device to other OSDs in the cluster is blocked. \\nEnsure that the cluster has available free space. It may be necessary to add capacity to the cluster to allow data from the failing device to successfully migrate, or to enable the balancer."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#device-health-in-use"\n summary: "Device failure is predicted, but unable to relocate data on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"DEVICE_HEALTH_IN_USE\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDFlapping"\n annotations:\n description: "OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked down and back up {{ $value | humanize }} times once a minute for 5 minutes. This may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster network, or the public network if no cluster network is deployed. Check the network stats on the listed host(s)."\n documentation: "https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds"\n summary: "Network issues are causing OSDs to flap (mark each other down) on cluster {{ $labels.cluster }}"\n expr: "(rate(ceph_osd_up[5m]) * on(cluster,ceph_daemon) group_left(hostname) ceph_osd_metadata) * 60 > 1"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.4"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDReadErrors"\n annotations:\n description: "An OSD has encountered read errors, but the OSD has recovered by retrying the reads. This may indicate an issue with hardware or the kernel."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#bluestore-spurious-read-errors"\n summary: "Device read errors detected on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"BLUESTORE_SPURIOUS_READ_ERRORS\\"} == 1"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPGImbalance"\n annotations:\n description: "OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} deviates by more than 30% from average PG count in the device class {{ $labels.device_class }}."\n summary: "PGs are not balanced across OSDs on cluster {{ $labels.cluster }}"\n expr: |\n abs(\n (\n (\n (ceph_osd_numpg > 0)\n * on (cluster, job, ceph_daemon) group_left(hostname, device_class) ceph_osd_metadata\n )\n - on (cluster, job, device_class) group_left avg(\n (ceph_osd_numpg > 0)\n * on (cluster, job, ceph_daemon) group_left(hostname, device_class) ceph_osd_metadata\n ) by (cluster, job, device_class)\n )\n / on (cluster, job, device_class) group_left avg(\n (ceph_osd_numpg > 0)\n * on (cluster, job, ceph_daemon) group_left(hostname, device_class) ceph_osd_metadata\n ) by (cluster, job, device_class)\n ) > 0.30\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.5"\n severity: "warning"\n type: "ceph_default"\n - name: "mds"\n rules:\n - alert: "CephFilesystemDamaged"\n annotations:\n description: "Filesystem metadata has been corrupted. Data may be inaccessible. Analyze metrics from the MDS daemon admin socket, or escalate to support."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages#cephfs-health-messages"\n summary: "CephFS filesystem is damaged on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"MDS_DAMAGE\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephFilesystemOffline"\n annotations:\n description: "All MDS ranks are unavailable. The MDS daemons managing metadata are down, rendering the filesystem offline."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-all-down"\n summary: "CephFS filesystem is offline on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"MDS_ALL_DOWN\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.3"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephFilesystemDegraded"\n annotations:\n description: "One or more metadata daemons (MDS ranks) are failed or in a damaged state. At best the filesystem is partially available, at worst the filesystem is completely unusable."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#fs-degraded"\n summary: "CephFS filesystem is degraded on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"FS_DEGRADED\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.4"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephFilesystemMDSRanksLow"\n annotations:\n description: "The filesystem\'s \'max_mds\' setting defines the number of MDS ranks in the filesystem. The current number of active MDS daemons is less than this value."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-up-less-than-max"\n summary: "Ceph MDS daemon count is lower than configured on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"MDS_UP_LESS_THAN_MAX\\"} > 0"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephFilesystemInsufficientStandby"\n annotations:\n description: "The minimum number of standby daemons required by standby_count_wanted is less than the current number of standby daemons. Adjust the standby count or increase the number of MDS daemons."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-insufficient-standby"\n summary: "Ceph filesystem standby daemons too few on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"MDS_INSUFFICIENT_STANDBY\\"} > 0"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephFilesystemFailureNoStandby"\n annotations:\n description: "An MDS daemon has failed, leaving only one active rank and no available standby. Investigate the cause of the failure or add a standby MDS."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#fs-with-failed-mds"\n summary: "MDS daemon failed, no further standby available on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"FS_WITH_FAILED_MDS\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.5"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephFilesystemReadOnly"\n annotations:\n description: "The filesystem has switched to READ ONLY due to an unexpected error when writing to the metadata pool. Either analyze the output from the MDS daemon admin socket, or escalate to support."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages#cephfs-health-messages"\n summary: "CephFS filesystem in read only mode due to write error(s) on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"MDS_HEALTH_READ_ONLY\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.2"\n severity: "critical"\n type: "ceph_default"\n - name: "mgr"\n rules:\n - alert: "CephMgrModuleCrash"\n annotations:\n description: "One or more mgr modules have crashed and have yet to be acknowledged by an administrator. A crashed module may impact functionality within the cluster. Use the \'ceph crash\' command to determine which module has failed, and archive it to acknowledge the failure."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#recent-mgr-module-crash"\n summary: "A manager module has recently crashed on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"RECENT_MGR_MODULE_CRASH\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.6.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephMgrPrometheusModuleInactive"\n annotations:\n description: "The mgr/prometheus module at {{ $labels.instance }} is unreachable. This could mean that the module has been disabled or the mgr daemon itself is down. Without the mgr/prometheus module metrics and alerts will no longer function. Open a shell to an admin node or toolbox pod and use \'ceph -s\' to to determine whether the mgr is active. If the mgr is not active, restart it, otherwise you can determine module status with \'ceph mgr module ls\'. If it is not listed as enabled, enable it with \'ceph mgr module enable prometheus\'."\n summary: "The mgr/prometheus module is not available"\n expr: "up{job=\\"ceph\\"} == 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.6.2"\n severity: "critical"\n type: "ceph_default"\n - name: "pgs"\n rules:\n - alert: "CephPGsInactive"\n annotations:\n description: "{{ $value }} PGs have been inactive for more than 5 minutes in pool {{ $labels.name }}. Inactive placement groups are not able to serve read/write requests."\n summary: "One or more placement groups are inactive on cluster {{ $labels.cluster }}"\n expr: "ceph_pool_metadata * on(cluster,pool_id,instance) group_left() (ceph_pg_total - ceph_pg_active) > 0"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGsUnclean"\n annotations:\n description: "{{ $value }} PGs have been unclean for more than 15 minutes in pool {{ $labels.name }}. Unclean PGs have not recovered from a previous failure."\n summary: "One or more placement groups are marked unclean on cluster {{ $labels.cluster }}"\n expr: "ceph_pool_metadata * on(cluster,pool_id,instance) group_left() (ceph_pg_total - ceph_pg_clean) > 0"\n for: "15m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.2"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPGsDamaged"\n annotations:\n description: "During data consistency checks (scrub), at least one PG has been flagged as being damaged or inconsistent. Check to see which PG is affected, and attempt a manual repair if necessary. To list problematic placement groups, use \'rados list-inconsistent-pg \'. To repair PGs use the \'ceph pg repair \' command."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-damaged"\n summary: "Placement group damaged, manual intervention needed on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=~\\"PG_DAMAGED|OSD_SCRUB_ERRORS\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.4"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGRecoveryAtRisk"\n annotations:\n description: "Data redundancy is at risk since one or more OSDs are at or above the \'full\' threshold. Add more capacity to the cluster, restore down/out OSDs, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-recovery-full"\n summary: "OSDs are too full for recovery on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"PG_RECOVERY_FULL\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.5"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGUnavailableBlockingIO"\n annotations:\n description: "Data availability is reduced, impacting the cluster\'s ability to service I/O. One or more placement groups (PGs) are in a state that blocks I/O."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-availability"\n summary: "PG is unavailable on cluster {{ $labels.cluster }}, blocking I/O"\n expr: "((ceph_health_detail{name=\\"PG_AVAILABILITY\\"} == 1) - scalar(ceph_health_detail{name=\\"OSD_DOWN\\"})) == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.3"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGBackfillAtRisk"\n annotations:\n description: "Data redundancy may be at risk due to lack of free space within the cluster. One or more OSDs have reached the \'backfillfull\' threshold. Add more capacity, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-backfill-full"\n summary: "Backfill operations are blocked due to lack of free space on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"PG_BACKFILL_FULL\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.6"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGNotScrubbed"\n annotations:\n description: "One or more PGs have not been scrubbed recently. Scrubs check metadata integrity, protecting against bit-rot. They check that metadata is consistent across data replicas. When PGs miss their scrub interval, it may indicate that the scrub window is too small, or PGs were not in a \'clean\' state during the scrub window. You can manually initiate a scrub with: ceph pg scrub "\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-not-scrubbed"\n summary: "Placement group(s) have not been scrubbed on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"PG_NOT_SCRUBBED\\"} == 1"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPGsHighPerOSD"\n annotations:\n description: "The number of placement groups per OSD is too high (exceeds the mon_max_pg_per_osd setting).\\n Check that the pg_autoscaler has not been disabled for any pools with \'ceph osd pool autoscale-status\', and that the profile selected is appropriate. You may also adjust the target_size_ratio of a pool to guide the autoscaler based on the expected relative size of the pool (\'ceph osd pool set cephfs.cephfs.meta target_size_ratio .1\') or set the pg_autoscaler mode to \'warn\' and adjust pg_num appropriately for one or more pools."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks/#too-many-pgs"\n summary: "Placement groups per OSD is too high on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"TOO_MANY_PGS\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPGNotDeepScrubbed"\n annotations:\n description: "One or more PGs have not been deep scrubbed recently. Deep scrubs protect against bit-rot. They compare data replicas to ensure consistency. When PGs miss their deep scrub interval, it may indicate that the window is too small or PGs were not in a \'clean\' state during the deep-scrub window."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-not-deep-scrubbed"\n summary: "Placement group(s) have not been deep scrubbed on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"PG_NOT_DEEP_SCRUBBED\\"} == 1"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "nodes"\n rules:\n - alert: "CephNodeRootFilesystemFull"\n annotations:\n description: "Root volume is dangerously full: {{ $value | humanize }}% free."\n summary: "Root filesystem is dangerously full"\n expr: "node_filesystem_avail_bytes{mountpoint=\\"/\\"} / node_filesystem_size_bytes{mountpoint=\\"/\\"} * 100 < 5"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.8.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephNodeNetworkPacketDrops"\n annotations:\n description: "Node {{ $labels.instance }} experiences packet drop > 0.5% or > 10 packets/s on interface {{ $labels.device }}."\n summary: "One or more NICs reports packet drops"\n expr: |\n (\n rate(node_network_receive_drop_total{device!="lo"}[1m]) +\n rate(node_network_transmit_drop_total{device!="lo"}[1m])\n ) / (\n rate(node_network_receive_packets_total{device!="lo"}[1m]) +\n rate(node_network_transmit_packets_total{device!="lo"}[1m])\n ) >= 0.0050000000000000001 and (\n rate(node_network_receive_drop_total{device!="lo"}[1m]) +\n rate(node_network_transmit_drop_total{device!="lo"}[1m])\n ) >= 10\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.8.2"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephNodeNetworkPacketErrors"\n annotations:\n description: "Node {{ $labels.instance }} experiences packet errors > 0.01% or > 10 packets/s on interface {{ $labels.device }}."\n summary: "One or more NICs reports packet errors on cluster {{ $labels.cluster }}"\n expr: |\n (\n rate(node_network_receive_errs_total{device!="lo"}[1m]) +\n rate(node_network_transmit_errs_total{device!="lo"}[1m])\n ) / (\n rate(node_network_receive_packets_total{device!="lo"}[1m]) +\n rate(node_network_transmit_packets_total{device!="lo"}[1m])\n ) >= 0.0001 or (\n rate(node_network_receive_errs_total{device!="lo"}[1m]) +\n rate(node_network_transmit_errs_total{device!="lo"}[1m])\n ) >= 10\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.8.3"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephNodeNetworkBondDegraded"\n annotations:\n description: "Bond {{ $labels.master }} is degraded on Node {{ $labels.instance }}."\n summary: "Degraded Bond on Node {{ $labels.instance }} on cluster {{ $labels.cluster }}"\n expr: |\n node_bonding_slaves - node_bonding_active != 0\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephNodeDiskspaceWarning"\n annotations:\n description: "Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will be full in less than 5 days based on the 48 hour trailing fill rate."\n summary: "Host filesystem free space is getting low on cluster {{ $labels.cluster }}"\n expr: "predict_linear(node_filesystem_free_bytes{device=~\\"/.*\\"}[2d], 3600 * 24 * 5) * on(cluster, instance) group_left(nodename) node_uname_info < 0"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.8.4"\n severity: "warning"\n type: "ceph_default"\n - alert: CephNodeInconsistentMTU\n expr: |\n node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)\n != on (cluster, device) group_left\n quantile by (cluster, device) (\n 0.5, node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)\n )\n labels:\n severity: warning\n type: ceph_default\n annotations:\n summary: "Node {{ $labels.instance }} has inconsistent MTU settings in cluster {{ $labels.cluster }}"\n description: "Network interface {{ $labels.device }} on node {{ $labels.instance }} has MTU {{ $value }} which differs from the cluster median."\n impact: |\n - May cause packet fragmentation or packet drops\n - Risk of degraded cluster communication and performance\n - Potential instability in services relying on consistent networking (e.g., Ceph, Kubernetes)\n fix: |\n - Check the MTU of interface `{{ $labels.device }}` on node `{{ $labels.instance }}`:\n ip link show {{ $labels.device }}\n\n - Find the median MTU value across the cluster by running this PromQL query in Prometheus:\n quantile by (cluster, device) (0.5, node_network_mtu_bytes * (node_network_up{device!="lo"} > 0))\n\n - Standardize MTU across all nodes to match the median (commonly 1500 or 9000):\n ip link set dev {{ $labels.device }} mtu \n\n - Make MTU setting persistent:\n - RHEL/CentOS: edit `/etc/sysconfig/network-scripts/ifcfg-`\n - Debian/Ubuntu: edit `/etc/netplan/*.yaml` and apply with `netplan apply`\n\n - Restart the affected interface or node if required.\n - name: "pools"\n rules:\n - alert: "CephPoolGrowthWarning"\n annotations:\n description: "Pool \'{{ $labels.name }}\' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours."\n summary: "Pool growth rate may soon exceed capacity on cluster {{ $labels.cluster }}"\n expr: "(predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(cluster,pool_id, instance) group_right() ceph_pool_metadata) >= 95"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.9.2"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPoolBackfillFull"\n annotations:\n description: "A pool is approaching the near full threshold, which will prevent recovery/backfill operations from completing. Consider adding more capacity."\n summary: "Free space in a pool is too low for recovery/backfill on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"POOL_BACKFILLFULL\\"} > 0"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPoolFull"\n annotations:\n description: "A pool has reached its MAX quota, or OSDs supporting the pool have reached the FULL threshold. Until this is resolved, writes to the pool will be blocked. Pool Breakdown (top 5) {{- range printf \\"topk(5, sort_desc(ceph_pool_percent_used{cluster=\'%s\'} * on(cluster,pool_id) group_right ceph_pool_metadata))\\" .Labels.cluster | query }} - {{ .Labels.name }} at {{ .Value }}% {{- end }} Increase the pool\'s quota, or add capacity to the cluster first then increase the pool\'s quota (e.g. ceph osd pool set quota max_bytes )"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pool-full"\n summary: "Pool is full - writes are blocked on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"POOL_FULL\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.9.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPoolNearFull"\n annotations:\n description: "A pool has exceeded the warning (percent full) threshold, or OSDs supporting the pool have reached the NEARFULL threshold. Writes may continue, but you are at risk of the pool going read-only if more capacity isn\'t made available. Determine the affected pool with \'ceph df detail\', looking at QUOTA BYTES and STORED. Increase the pool\'s quota, or add capacity to the cluster first then increase the pool\'s quota (e.g. ceph osd pool set quota max_bytes ). Also ensure that the balancer is active."\n summary: "One or more Ceph pools are nearly full on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"POOL_NEAR_FULL\\"} > 0"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "healthchecks"\n rules:\n - alert: "CephSlowOps"\n annotations:\n description: "{{ $value }} OSD requests are taking too long to process (osd_op_complaint_time exceeded)"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#slow-ops"\n summary: "OSD operations are slow to complete on cluster {{ $labels.cluster }}"\n expr: "ceph_healthcheck_slow_ops > 0"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephDaemonSlowOps"\n annotations:\n description: "{{ $labels.ceph_daemon }} operations are taking too long to process (complaint time exceeded)"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#slow-ops"\n summary: "{{ $labels.ceph_daemon }} operations are slow to complete on cluster {{ $labels.cluster }}"\n expr: "ceph_daemon_health_metrics{type=\\"SLOW_OPS\\"} > 0"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "cephadm"\n rules:\n - alert: "CephadmUpgradeFailed"\n annotations:\n description: "The cephadm cluster upgrade process has failed. The cluster remains in an undetermined state. Please review the cephadm logs, to understand the nature of the issue"\n summary: "Ceph version upgrade has failed on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"UPGRADE_EXCEPTION\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.11.2"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephadmDaemonFailed"\n annotations:\n description: "A daemon managed by cephadm is no longer active. Determine, which daemon is down with \'ceph health detail\'. you may start daemons with the \'ceph orch daemon start \'"\n summary: "A ceph daemon managed by cephadm is down on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"CEPHADM_FAILED_DAEMON\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.11.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephadmPaused"\n annotations:\n description: "Cluster management has been paused manually. This will prevent the orchestrator from service management and reconciliation. If this is not intentional, resume cephadm operations with \'ceph orch resume\'"\n documentation: "https://docs.ceph.com/en/latest/cephadm/operations#cephadm-paused"\n summary: "Orchestration tasks via cephadm are PAUSED on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"CEPHADM_PAUSED\\"} > 0"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "hardware"\n rules:\n - alert: "HardwareStorageError"\n annotations:\n description: "Some storage devices are in error. Check `ceph health detail`."\n summary: "Storage devices error(s) detected on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"HARDWARE_STORAGE\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwareMemoryError"\n annotations:\n description: "DIMM error(s) detected. Check `ceph health detail`."\n summary: "DIMM error(s) detected on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"HARDWARE_MEMORY\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.2"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwareProcessorError"\n annotations:\n description: "Processor error(s) detected. Check `ceph health detail`."\n summary: "Processor error(s) detected on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"HARDWARE_PROCESSOR\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.3"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwareNetworkError"\n annotations:\n description: "Network error(s) detected. Check `ceph health detail`."\n summary: "Network error(s) detected on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"HARDWARE_NETWORK\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.4"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwarePowerError"\n annotations:\n description: "Power supply error(s) detected. Check `ceph health detail`."\n summary: "Power supply error(s) detected on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"HARDWARE_POWER\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.5"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwareFanError"\n annotations:\n description: "Fan error(s) detected. Check `ceph health detail`."\n summary: "Fan error(s) detected on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"HARDWARE_FANS\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.6"\n severity: "critical"\n type: "ceph_default"\n - name: "PrometheusServer"\n rules:\n - alert: "PrometheusJobMissing"\n annotations:\n description: "The prometheus job that scrapes from Ceph is no longer defined, this will effectively mean you\'ll have no metrics or alerts for the cluster. Please review the job definitions in the prometheus.yml file of the prometheus instance."\n summary: "The scrape job for Ceph is missing from Prometheus"\n expr: "absent(up{job=\\"ceph\\"})"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.12.1"\n severity: "critical"\n type: "ceph_default"\n - name: "rados"\n rules:\n - alert: "CephObjectMissing"\n annotations:\n description: "The latest version of a RADOS object can not be found, even though all OSDs are up. I/O requests for this object from clients will block (hang). Resolving this issue may require the object to be rolled back to a prior version manually, and manually verified."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#object-unfound"\n summary: "Object(s) marked UNFOUND on cluster {{ $labels.cluster }}"\n expr: "(ceph_health_detail{name=\\"OBJECT_UNFOUND\\"} == 1) * on() group_right(cluster) (count(ceph_osd_up == 1) by (cluster) == bool count(ceph_osd_metadata) by(cluster)) == 1"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.1"\n severity: "critical"\n type: "ceph_default"\n - name: "generic"\n rules:\n - alert: "CephDaemonCrash"\n annotations:\n description: "One or more daemons have crashed recently, and need to be acknowledged. This notification ensures that software crashes do not go unseen. To acknowledge a crash, use the \'ceph crash archive \' command."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks/#recent-crash"\n summary: "One or more Ceph daemons have crashed, and are pending acknowledgement on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"RECENT_CRASH\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.1.2"\n severity: "critical"\n type: "ceph_default"\n - name: "rbdmirror"\n rules:\n - alert: "CephRBDMirrorImagesPerDaemonHigh"\n annotations:\n description: "Number of image replications per daemon is not supposed to go beyond threshold 100"\n summary: "Number of image replications are now above 100 on cluster {{ $labels.cluster }}"\n expr: "sum by (cluster, ceph_daemon, namespace) (ceph_rbd_mirror_snapshot_image_snapshots) > 100"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.2"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephRBDMirrorImagesNotInSync"\n annotations:\n description: "Both local and remote RBD mirror images should be in sync."\n summary: "Some of the RBD mirror images are not in sync with the remote counter parts on cluster {{ $labels.cluster }}"\n expr: "sum by (cluster, ceph_daemon, image, namespace, pool) (topk by (cluster, ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_local_timestamp) - topk by (cluster, ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_remote_timestamp)) != 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.3"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephRBDMirrorImagesNotInSyncVeryHigh"\n annotations:\n description: "More than 10% of the images have synchronization problems."\n summary: "Number of unsynchronized images are very high on cluster {{ $labels.cluster }}"\n expr: "count by (ceph_daemon, cluster) ((topk by (cluster, ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_local_timestamp) - topk by (cluster, ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_remote_timestamp)) != 0) > (sum by (ceph_daemon, cluster) (ceph_rbd_mirror_snapshot_snapshots)*.1)"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.4"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephRBDMirrorImageTransferBandwidthHigh"\n annotations:\n description: "Detected a heavy increase in bandwidth for rbd replications (over 80%) in the last 30 min. This might not be a problem, but it is good to review the number of images being replicated simultaneously"\n summary: "The replication network usage on cluster {{ $labels.cluster }} has been increased over 80% in the last 30 minutes. Review the number of images being replicated. This alert will be cleaned automatically after 30 minutes"\n expr: "rate(ceph_rbd_mirror_journal_replay_bytes[30m]) > 0.80"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.5"\n severity: "warning"\n type: "ceph_default"\n - name: "nvmeof"\n rules:\n - alert: "NVMeoFSubsystemNamespaceLimit"\n annotations:\n description: "Subsystems have a max namespace limit defined at creation time. This alert means that no more namespaces can be added to {{ $labels.nqn }}"\n summary: "{{ $labels.nqn }} subsystem has reached its maximum number of namespaces on cluster {{ $labels.cluster }}"\n expr: "(count by(nqn, cluster, instance) (ceph_nvmeof_subsystem_namespace_metadata)) >= on(nqn, instance) group_right(cluster) ceph_nvmeof_subsystem_namespace_limit"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFMultipleNamespacesOfRBDImage"\n annotations:\n description: "Each NVMeoF namespace must have a unique RBD pool and image, across all different gateway groups."\n summary: "RBD image {{ $labels.pool_name }}/{{ $labels.rbd_name }} cannot be reused for multiple NVMeoF namespace "\n expr: "count by(pool_name, rbd_name) (count by(bdev_name, pool_name, rbd_name) (ceph_nvmeof_bdev_metadata and on (bdev_name) ceph_nvmeof_subsystem_namespace_metadata)) > 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFTooManyGateways"\n annotations:\n description: "You may create many gateways, but 32 is the tested limit"\n summary: "Max supported gateways exceeded on cluster {{ $labels.cluster }}"\n expr: "count(ceph_nvmeof_gateway_info) by (cluster) > 32.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFMaxGatewayGroupSize"\n annotations:\n description: "You may create many gateways in a gateway group, but 8 is the tested limit"\n summary: "Max gateways within a gateway group ({{ $labels.group }}) exceeded on cluster {{ $labels.cluster }}"\n expr: "count(ceph_nvmeof_gateway_info) by (cluster,group) > 8.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFMaxGatewayGroups"\n annotations:\n description: "You may create many gateway groups, but 4 is the tested limit"\n summary: "Max gateway groups exceeded on cluster {{ $labels.cluster }}"\n expr: "count(count by (group, cluster) (ceph_nvmeof_gateway_info)) by (cluster) > 4.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFSingleGateway"\n annotations:\n description: "Although a single member gateway group is valid, it should only be used for test purposes"\n summary: "The gateway group {{ $labels.group }} consists of a single gateway - HA is not possible on cluster {{ $labels.cluster }}"\n expr: "count(ceph_nvmeof_gateway_info) by(cluster,group) == 1"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighGatewayCPU"\n annotations:\n description: "Typically, high CPU may indicate degraded performance. Consider increasing the number of reactor cores"\n summary: "CPU used by {{ $labels.instance }} NVMe-oF Gateway is high on cluster {{ $labels.cluster }}"\n expr: "label_replace(avg by(instance, cluster) (rate(ceph_nvmeof_reactor_seconds_total{mode=\\"busy\\"}[1m])),\\"instance\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\") > 80.00"\n for: "10m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFGatewayOpenSecurity"\n annotations:\n description: "It is good practice to ensure subsystems use host security to reduce the risk of unexpected data loss"\n summary: "Subsystem {{ $labels.nqn }} has been defined without host level security on cluster {{ $labels.cluster }}"\n expr: "ceph_nvmeof_subsystem_metadata{allow_any_host=\\"yes\\"}"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFTooManySubsystems"\n annotations:\n description: "NVMeoF gateway {{ $labels.gateway_host }} has reached or exceeded the supported maximum of 128 subsystems. Current count: {{ $value }}."\n summary: "The number of subsystems defined to the NVMeoF gateway reached or exceeded the supported values on cluster {{ $labels.cluster }}"\n expr: "count by(gateway_host, cluster) (label_replace(ceph_nvmeof_subsystem_metadata,\\"gateway_host\\",\\"$1\\",\\"instance\\",\\"(.*?)(?::.*)?\\")) >= 128.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFTooManyNamespaces"\n annotations:\n description: "NVMeoF gateway {{ $labels.gateway_host }} has reached or exceeded the supported maximum of 4096 namespaces. Current count: {{ $value }}."\n summary: "The number of namespaces defined to the NVMeoF gateway reached or exceeded supported values on cluster {{ $labels.cluster }}"\n expr: "sum by(gateway_host, cluster) (label_replace(ceph_nvmeof_subsystem_namespace_count,\\"gateway_host\\",\\"$1\\",\\"instance\\",\\"(.*?)(?::.*)?\\")) >= 4096.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFVersionMismatch"\n annotations:\n description: "This may indicate an issue with deployment. Check cephadm logs"\n summary: "Too many different NVMe-oF gateway releases active on cluster {{ $labels.cluster }}"\n expr: "count(count(ceph_nvmeof_gateway_info) by (cluster, version)) by (cluster) > 1"\n for: "1h"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighClientCount"\n annotations:\n description: "The supported limit for clients connecting to a subsystem is 128"\n summary: "The number of clients connected to {{ $labels.nqn }} is too high on cluster {{ $labels.cluster }}"\n expr: "ceph_nvmeof_subsystem_host_count > 128.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFMissingListener"\n annotations:\n description: "For every subsystem, each gateway should have a listener to balance traffic between gateways."\n summary: "No listener added for {{ $labels.instance }} NVMe-oF Gateway to {{ $labels.nqn }} subsystem"\n expr: "ceph_nvmeof_subsystem_listener_count == 0 and on(nqn) sum(ceph_nvmeof_subsystem_listener_count) by (nqn) > 0"\n for: "10m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFZeroListenerSubsystem"\n annotations:\n description: "NVMeoF gateway configuration incomplete; one of the subsystems have zero listeners."\n summary: "No listeners added to {{ $labels.nqn }} subsystem"\n expr: "sum(ceph_nvmeof_subsystem_listener_count) by (nqn) == 0"\n for: "10m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighHostCPU"\n annotations:\n description: "High CPU on a gateway host can lead to CPU contention and performance degradation"\n summary: "The CPU is high ({{ $value }}%) on NVMeoF Gateway host ({{ $labels.host }}) on cluster {{ $labels.cluster }}"\n expr: "100-((100*(avg by(cluster,host) (label_replace(rate(node_cpu_seconds_total{mode=\\"idle\\"}[5m]),\\"host\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\")) * on(cluster, host) group_right label_replace(ceph_nvmeof_gateway_info,\\"host\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\")))) >= 80.00"\n for: "10m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFInterfaceDown"\n annotations:\n description: "A NIC used by one or more subsystems is in a down state"\n summary: "Network interface {{ $labels.device }} is down on cluster {{ $labels.cluster }}"\n expr: "ceph_nvmeof_subsystem_listener_iface_info{operstate=\\"down\\"}"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.14.1"\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFInterfaceDuplex"\n annotations:\n description: "Until this is resolved, performance from the gateway will be degraded"\n summary: "Network interface {{ $labels.device }} is not running in full duplex mode on cluster {{ $labels.cluster }}"\n expr: "ceph_nvmeof_subsystem_listener_iface_info{duplex!=\\"full\\"}"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighReadLatency"\n annotations:\n description: "High latencies may indicate a constraint within the cluster e.g. CPU, network. Please investigate"\n summary: "The average read latency over the last 5 mins has reached 10 ms or more on {{ $labels.gateway }}"\n expr: "label_replace((avg by(instance) ((rate(ceph_nvmeof_bdev_read_seconds_total[1m]) / rate(ceph_nvmeof_bdev_reads_completed_total[1m])))),\\"gateway\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\") > 0.01"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighWriteLatency"\n annotations:\n description: "High latencies may indicate a constraint within the cluster e.g. CPU, network. Please investigate"\n summary: "The average write latency over the last 5 mins has reached 20 ms or more on {{ $labels.gateway }}"\n expr: "label_replace((avg by(instance) ((rate(ceph_nvmeof_bdev_write_seconds_total[5m]) / rate(ceph_nvmeof_bdev_writes_completed_total[5m])))),\\"gateway\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\") > 0.02"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHostKeepAliveTimeout"\n annotations:\n description: "Host was disconnected due to host keep alive timeout"\n summary: "Host ({{ $labels.host_nqn }}) was disconnected {{ $value }} times from subsystem ({{ $labels.nqn }}) in last 24 hours"\n expr: "ceil(changes(ceph_nvmeof_host_keepalive_timeout[24h:]) / 2) > 0"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n', '/etc/prometheus/alerting/custom_alerts.yml': ''}, 'retention_time': '15d', 'retention_size': '0', 'ip_to_bind_to': '', 'use_url_prefix': False}} 2026-02-14 01:36:45,390 7f9d5b64ce00 DEBUG Determined image: 'quay.io/prometheus/prometheus:v2.51.0' 2026-02-14 01:36:45,404 7f9d5b64ce00 INFO Reconfig daemon prometheus.trial014 ... 2026-02-14 01:36:45,586 7f9d5b64ce00 DEBUG firewalld.service is not enabled 2026-02-14 01:36:45,586 7f9d5b64ce00 DEBUG Not possible to open ports <[9095]>. firewalld.service is not available 2026-02-14 01:36:46,998 7fc830e22e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'inspect-image'] 2026-02-14 01:36:47,035 7fc830e22e00 DEBUG /usr/bin/podman: stdout edc95c9460f10a90f1e921f9347538ed9be0b1c7aee54201d89be4c7dacb23c7,[quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9] 2026-02-14 01:36:47,165 7fc830e22e00 DEBUG ceph: stdout ceph version 20.2.0-667-geae49298 (eae49298735c216a37f3957204b0fe058b58e376) tentacle (stable) 2026-02-14 01:36:47,304 7f020fe90e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:36:47,320 7f020fe90e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'mgr.trial014.uucrmp', 'image': '', 'deploy_arguments': [], 'params': {'tcp_ports': [8443, 9283, 8765], 'allow_ptrace': True}, 'meta': {'service_name': 'mgr', 'ports': [8443, 9283, 8765], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[mgr.trial014.uucrmp]\nkey = AQDz0I9pYGlEFhAAOnF6ryAzuu/H9SeCFjrELQ==\n'}} 2026-02-14 01:36:47,320 7f020fe90e00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9' 2026-02-14 01:36:47,334 7f020fe90e00 INFO Redeploy daemon mgr.trial014.uucrmp ... 2026-02-14 01:36:48,126 7f020fe90e00 DEBUG firewalld.service is not enabled 2026-02-14 01:36:48,126 7f020fe90e00 DEBUG Not possible to enable service . firewalld.service is not available 2026-02-14 01:36:48,140 7f020fe90e00 DEBUG firewalld.service is not enabled 2026-02-14 01:36:48,141 7f020fe90e00 DEBUG Not possible to open ports <[8443, 9283, 8765]>. firewalld.service is not available 2026-02-14 01:36:48,283 7f27051c3e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ls'] 2026-02-14 01:36:50,126 7f2f167fde00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:36:50,362 7f8fc75d5e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'list-networks'] 2026-02-14 01:36:50,522 7f1fc04bce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 01:36:50,538 7f1fc04bce00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:36:50,539 7f1fc04bce00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:36:52,043 7f53025dee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ls'] 2026-02-14 01:36:53,190 7fe7ba5e9e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:36:53,430 7fe76068de00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'list-networks'] 2026-02-14 01:36:54,311 7fa365c09e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.io/prometheus/prometheus:v2.51.0', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:36:54,328 7fa365c09e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'prometheus.trial014', 'image': '', 'deploy_arguments': [], 'params': {'tcp_ports': [9095], 'reconfig': True, 'allow_ptrace': True}, 'meta': {'service_name': 'prometheus', 'ports': [9095], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'files': {'prometheus.yml': "# This file is generated by cephadm.\nglobal:\n scrape_interval: 10s\n evaluation_interval: 10s\n external_labels:\n cluster: 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\nrule_files:\n - /etc/prometheus/alerting/*\n\nalerting:\n alertmanagers:\n - scheme: http\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=alertmanager\n\nscrape_configs:\n - job_name: 'ceph'\n relabel_configs:\n - source_labels: [__address__]\n target_label: cluster\n replacement: 29ce0a10-0945-11f1-b2be-d404e6e7d460\n - source_labels: [instance]\n target_label: instance\n replacement: 'ceph_cluster'\n honor_labels: true\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=mgr-prometheus\n\n - job_name: 'node'\n relabel_configs:\n - source_labels: [__address__]\n target_label: cluster\n replacement: 29ce0a10-0945-11f1-b2be-d404e6e7d460\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=node-exporter\n\n\n - job_name: 'ceph-exporter'\n relabel_configs:\n - source_labels: [__address__]\n target_label: cluster\n replacement: 29ce0a10-0945-11f1-b2be-d404e6e7d460\n honor_labels: true\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=ceph-exporter\n\n - job_name: 'nvmeof'\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=nvmeof\n\n - job_name: 'nfs'\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=nfs\n\n - job_name: 'smb'\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=smb\n\n", '/etc/prometheus/alerting/ceph_alerts.yml': 'groups:\n - name: "cluster health"\n rules:\n - alert: "CephHealthError"\n annotations:\n description: "The cluster state has been HEALTH_ERROR for more than 5 minutes on cluster {{ $labels.cluster }}. Please check \'ceph health detail\' for more information."\n summary: "Ceph is in the ERROR state on cluster {{ $labels.cluster }}"\n expr: "ceph_health_status == 2"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.2.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephHealthWarning"\n annotations:\n description: "The cluster state has been HEALTH_WARN for more than 15 minutes on cluster {{ $labels.cluster }}. Please check \'ceph health detail\' for more information."\n summary: "Ceph is in the WARNING state on cluster {{ $labels.cluster }}"\n expr: "ceph_health_status == 1"\n for: "15m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "mon"\n rules:\n - alert: "CephMonDownQuorumAtRisk"\n annotations:\n description: "{{ $min := printf \\"floor(count(ceph_mon_metadata{cluster=\'%s\'}) / 2) + 1\\" .Labels.cluster | query | first | value }}Quorum requires a majority of monitors (x {{ $min }}) to be active. Without quorum the cluster will become inoperable, affecting all services and connected clients. The following monitors are down: {{- range printf \\"(ceph_mon_quorum_status{cluster=\'%s\'} == 0) + on(cluster,ceph_daemon) group_left(hostname) (ceph_mon_metadata * 0)\\" .Labels.cluster | query }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-down"\n summary: "Monitor quorum is at risk on cluster {{ $labels.cluster }}"\n expr: |\n (\n (ceph_health_detail{name="MON_DOWN"} == 1) * on() group_right(cluster) (\n count(ceph_mon_quorum_status == 1) by(cluster)== bool (floor(count(ceph_mon_metadata) by(cluster) / 2) + 1)\n )\n ) == 1\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.3.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephMonDown"\n annotations:\n description: "{{ $down := printf \\"count(ceph_mon_quorum_status{cluster=\'%s\'} == 0)\\" .Labels.cluster | query | first | value }}{{ $s := \\"\\" }}{{ if gt $down 1.0 }}{{ $s = \\"s\\" }}{{ end }}You have {{ $down }} monitor{{ $s }} down. Quorum is still intact, but the loss of an additional monitor will make your cluster inoperable. The following monitors are down: {{- range printf \\"(ceph_mon_quorum_status{cluster=\'%s\'} == 0) + on(cluster,ceph_daemon) group_left(hostname) (ceph_mon_metadata * 0)\\" .Labels.cluster | query }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-down"\n summary: "One or more monitors down on cluster {{ $labels.cluster }}"\n expr: |\n (count by (cluster) (ceph_mon_quorum_status == 0)) <= (count by (cluster) (ceph_mon_metadata) - floor((count by (cluster) (ceph_mon_metadata) / 2 + 1)))\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephMonDiskspaceCritical"\n annotations:\n description: "The free space available to a monitor\'s store is critically low. You should increase the space available to the monitor(s). The default directory is /var/lib/ceph/mon-*/data/store.db on traditional deployments, and /var/lib/rook/mon-*/data/store.db on the mon pod\'s worker node for Rook. Look for old, rotated versions of *.log and MANIFEST*. Do NOT touch any *.sst files. Also check any other directories under /var/lib/rook and other directories on the same filesystem, often /var/log and /var/tmp are culprits. Your monitor hosts are; {{- range query \\"ceph_mon_metadata\\"}} - {{ .Labels.hostname }} {{- end }}"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-disk-crit"\n summary: "Filesystem space on at least one monitor is critically low on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"MON_DISK_CRIT\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.3.2"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephMonDiskspaceLow"\n annotations:\n description: "The space available to a monitor\'s store is approaching full (>70% is the default). You should increase the space available to the monitor(s). The default directory is /var/lib/ceph/mon-*/data/store.db on traditional deployments, and /var/lib/rook/mon-*/data/store.db on the mon pod\'s worker node for Rook. Look for old, rotated versions of *.log and MANIFEST*. Do NOT touch any *.sst files. Also check any other directories under /var/lib/rook and other directories on the same filesystem, often /var/log and /var/tmp are culprits. Your monitor hosts are; {{- range query \\"ceph_mon_metadata\\"}} - {{ .Labels.hostname }} {{- end }}"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-disk-low"\n summary: "Drive space on at least one monitor is approaching full on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"MON_DISK_LOW\\"} == 1"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephMonClockSkew"\n annotations:\n description: "Ceph monitors rely on closely synchronized time to maintain quorum and cluster consistency. This event indicates that the time on at least one mon has drifted too far from the lead mon. Review cluster status with ceph -s. This will show which monitors are affected. Check the time sync status on each monitor host with \'ceph time-sync-status\' and the state and peers of your ntpd or chrony daemon."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-clock-skew"\n summary: "Clock skew detected among monitors on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"MON_CLOCK_SKEW\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "osd"\n rules:\n - alert: "CephOSDDownHigh"\n annotations:\n description: "{{ $value | humanize }}% or {{ with printf \\"count (ceph_osd_up{cluster=\'%s\'} == 0)\\" .Labels.cluster | query }}{{ . | first | value }}{{ end }} of {{ with printf \\"count (ceph_osd_up{cluster=\'%s\'})\\" .Labels.cluster | query }}{{ . | first | value }}{{ end }} OSDs are down (>= 10%). The following OSDs are down: {{- range printf \\"(ceph_osd_up{cluster=\'%s\'} * on(cluster, ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0\\" .Labels.cluster | query }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}"\n summary: "More than 10% of OSDs are down on cluster {{ $labels.cluster }}"\n expr: "count by (cluster) (ceph_osd_up == 0) / count by (cluster) (ceph_osd_up) * 100 >= 10"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephOSDHostDown"\n annotations:\n description: "The following OSDs are down: {{- range printf \\"(ceph_osd_up{cluster=\'%s\'} * on(cluster,ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0\\" .Labels.cluster | query }} - {{ .Labels.hostname }} : {{ .Labels.ceph_daemon }} {{- end }}"\n summary: "An OSD host is offline on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"OSD_HOST_DOWN\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.8"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDDown"\n annotations:\n description: "{{ $num := printf \\"count(ceph_osd_up{cluster=\'%s\'} == 0) \\" .Labels.cluster | query | first | value }}{{ $s := \\"\\" }}{{ if gt $num 1.0 }}{{ $s = \\"s\\" }}{{ end }}{{ $num }} OSD{{ $s }} down for over 5mins. The following OSD{{ $s }} {{ if eq $s \\"\\" }}is{{ else }}are{{ end }} down: {{- range printf \\"(ceph_osd_up{cluster=\'%s\'} * on(cluster,ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0\\" .Labels.cluster | query }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-down"\n summary: "An OSD has been marked down on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"OSD_DOWN\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.2"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDNearFull"\n annotations:\n description: "One or more OSDs have reached the NEARFULL threshold. Use \'ceph health detail\' and \'ceph osd df\' to identify the problem. To resolve, add capacity to the affected OSD\'s failure domain, restore down/out OSDs, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-nearfull"\n summary: "OSD(s) running low on free space (NEARFULL) on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"OSD_NEARFULL\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.3"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDFull"\n annotations:\n description: "An OSD has reached the FULL threshold. Writes to pools that share the affected OSD will be blocked. Use \'ceph health detail\' and \'ceph osd df\' to identify the problem. To resolve, add capacity to the affected OSD\'s failure domain, restore down/out OSDs, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-full"\n summary: "OSD full, writes blocked on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"OSD_FULL\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.6"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephOSDBackfillFull"\n annotations:\n description: "An OSD has reached the BACKFILL FULL threshold. This will prevent rebalance operations from completing. Use \'ceph health detail\' and \'ceph osd df\' to identify the problem. To resolve, add capacity to the affected OSD\'s failure domain, restore down/out OSDs, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-backfillfull"\n summary: "OSD(s) too full for backfill operations on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"OSD_BACKFILLFULL\\"} > 0"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDTooManyRepairs"\n annotations:\n description: "Reads from an OSD have used a secondary PG to return data to the client, indicating a potential failing drive."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-too-many-repairs"\n summary: "OSD reports a high number of read errors on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"OSD_TOO_MANY_REPAIRS\\"} == 1"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDTimeoutsPublicNetwork"\n annotations:\n description: "OSD heartbeats on the cluster\'s \'public\' network (frontend) are running slow. Investigate the network for latency or loss issues. Use \'ceph health detail\' to show the affected OSDs."\n summary: "Network issues delaying OSD heartbeats (public network) on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"OSD_SLOW_PING_TIME_FRONT\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDTimeoutsClusterNetwork"\n annotations:\n description: "OSD heartbeats on the cluster\'s \'cluster\' network (backend) are slow. Investigate the network for latency issues on this subnet. Use \'ceph health detail\' to show the affected OSDs."\n summary: "Network issues delaying OSD heartbeats (cluster network) on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"OSD_SLOW_PING_TIME_BACK\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDInternalDiskSizeMismatch"\n annotations:\n description: "One or more OSDs have an internal inconsistency between metadata and the size of the device. This could lead to the OSD(s) crashing in future. You should redeploy the affected OSDs."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#bluestore-disk-size-mismatch"\n summary: "OSD size inconsistency error on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"BLUESTORE_DISK_SIZE_MISMATCH\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephDeviceFailurePredicted"\n annotations:\n description: "The device health module has determined that one or more devices will fail soon. To review device status use \'ceph device ls\'. To show a specific device use \'ceph device info \'. Mark the OSD out so that data may migrate to other OSDs. Once the OSD has drained, destroy the OSD, replace the device, and redeploy the OSD."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#id2"\n summary: "Device(s) predicted to fail soon on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"DEVICE_HEALTH\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephDeviceFailurePredictionTooHigh"\n annotations:\n description: "The device health module has determined that devices predicted to fail can not be remediated automatically, since too many OSDs would be removed from the cluster to ensure performance and availability. Prevent data integrity issues by adding new OSDs so that data may be relocated."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#device-health-toomany"\n summary: "Too many devices are predicted to fail on cluster {{ $labels.cluster }}, unable to resolve"\n expr: "ceph_health_detail{name=\\"DEVICE_HEALTH_TOOMANY\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.7"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephDeviceFailureRelocationIncomplete"\n annotations:\n description: "The device health module has determined that one or more devices will fail soon, but the normal process of relocating the data on the device to other OSDs in the cluster is blocked. \\nEnsure that the cluster has available free space. It may be necessary to add capacity to the cluster to allow data from the failing device to successfully migrate, or to enable the balancer."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#device-health-in-use"\n summary: "Device failure is predicted, but unable to relocate data on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"DEVICE_HEALTH_IN_USE\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDFlapping"\n annotations:\n description: "OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked down and back up {{ $value | humanize }} times once a minute for 5 minutes. This may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster network, or the public network if no cluster network is deployed. Check the network stats on the listed host(s)."\n documentation: "https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds"\n summary: "Network issues are causing OSDs to flap (mark each other down) on cluster {{ $labels.cluster }}"\n expr: "(rate(ceph_osd_up[5m]) * on(cluster,ceph_daemon) group_left(hostname) ceph_osd_metadata) * 60 > 1"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.4"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDReadErrors"\n annotations:\n description: "An OSD has encountered read errors, but the OSD has recovered by retrying the reads. This may indicate an issue with hardware or the kernel."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#bluestore-spurious-read-errors"\n summary: "Device read errors detected on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"BLUESTORE_SPURIOUS_READ_ERRORS\\"} == 1"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPGImbalance"\n annotations:\n description: "OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} deviates by more than 30% from average PG count in the device class {{ $labels.device_class }}."\n summary: "PGs are not balanced across OSDs on cluster {{ $labels.cluster }}"\n expr: |\n abs(\n (\n (\n (ceph_osd_numpg > 0)\n * on (cluster, job, ceph_daemon) group_left(hostname, device_class) ceph_osd_metadata\n )\n - on (cluster, job, device_class) group_left avg(\n (ceph_osd_numpg > 0)\n * on (cluster, job, ceph_daemon) group_left(hostname, device_class) ceph_osd_metadata\n ) by (cluster, job, device_class)\n )\n / on (cluster, job, device_class) group_left avg(\n (ceph_osd_numpg > 0)\n * on (cluster, job, ceph_daemon) group_left(hostname, device_class) ceph_osd_metadata\n ) by (cluster, job, device_class)\n ) > 0.30\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.5"\n severity: "warning"\n type: "ceph_default"\n - name: "mds"\n rules:\n - alert: "CephFilesystemDamaged"\n annotations:\n description: "Filesystem metadata has been corrupted. Data may be inaccessible. Analyze metrics from the MDS daemon admin socket, or escalate to support."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages#cephfs-health-messages"\n summary: "CephFS filesystem is damaged on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"MDS_DAMAGE\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephFilesystemOffline"\n annotations:\n description: "All MDS ranks are unavailable. The MDS daemons managing metadata are down, rendering the filesystem offline."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-all-down"\n summary: "CephFS filesystem is offline on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"MDS_ALL_DOWN\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.3"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephFilesystemDegraded"\n annotations:\n description: "One or more metadata daemons (MDS ranks) are failed or in a damaged state. At best the filesystem is partially available, at worst the filesystem is completely unusable."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#fs-degraded"\n summary: "CephFS filesystem is degraded on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"FS_DEGRADED\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.4"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephFilesystemMDSRanksLow"\n annotations:\n description: "The filesystem\'s \'max_mds\' setting defines the number of MDS ranks in the filesystem. The current number of active MDS daemons is less than this value."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-up-less-than-max"\n summary: "Ceph MDS daemon count is lower than configured on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"MDS_UP_LESS_THAN_MAX\\"} > 0"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephFilesystemInsufficientStandby"\n annotations:\n description: "The minimum number of standby daemons required by standby_count_wanted is less than the current number of standby daemons. Adjust the standby count or increase the number of MDS daemons."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-insufficient-standby"\n summary: "Ceph filesystem standby daemons too few on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"MDS_INSUFFICIENT_STANDBY\\"} > 0"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephFilesystemFailureNoStandby"\n annotations:\n description: "An MDS daemon has failed, leaving only one active rank and no available standby. Investigate the cause of the failure or add a standby MDS."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#fs-with-failed-mds"\n summary: "MDS daemon failed, no further standby available on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"FS_WITH_FAILED_MDS\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.5"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephFilesystemReadOnly"\n annotations:\n description: "The filesystem has switched to READ ONLY due to an unexpected error when writing to the metadata pool. Either analyze the output from the MDS daemon admin socket, or escalate to support."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages#cephfs-health-messages"\n summary: "CephFS filesystem in read only mode due to write error(s) on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"MDS_HEALTH_READ_ONLY\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.2"\n severity: "critical"\n type: "ceph_default"\n - name: "mgr"\n rules:\n - alert: "CephMgrModuleCrash"\n annotations:\n description: "One or more mgr modules have crashed and have yet to be acknowledged by an administrator. A crashed module may impact functionality within the cluster. Use the \'ceph crash\' command to determine which module has failed, and archive it to acknowledge the failure."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#recent-mgr-module-crash"\n summary: "A manager module has recently crashed on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"RECENT_MGR_MODULE_CRASH\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.6.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephMgrPrometheusModuleInactive"\n annotations:\n description: "The mgr/prometheus module at {{ $labels.instance }} is unreachable. This could mean that the module has been disabled or the mgr daemon itself is down. Without the mgr/prometheus module metrics and alerts will no longer function. Open a shell to an admin node or toolbox pod and use \'ceph -s\' to to determine whether the mgr is active. If the mgr is not active, restart it, otherwise you can determine module status with \'ceph mgr module ls\'. If it is not listed as enabled, enable it with \'ceph mgr module enable prometheus\'."\n summary: "The mgr/prometheus module is not available"\n expr: "up{job=\\"ceph\\"} == 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.6.2"\n severity: "critical"\n type: "ceph_default"\n - name: "pgs"\n rules:\n - alert: "CephPGsInactive"\n annotations:\n description: "{{ $value }} PGs have been inactive for more than 5 minutes in pool {{ $labels.name }}. Inactive placement groups are not able to serve read/write requests."\n summary: "One or more placement groups are inactive on cluster {{ $labels.cluster }}"\n expr: "ceph_pool_metadata * on(cluster,pool_id,instance) group_left() (ceph_pg_total - ceph_pg_active) > 0"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGsUnclean"\n annotations:\n description: "{{ $value }} PGs have been unclean for more than 15 minutes in pool {{ $labels.name }}. Unclean PGs have not recovered from a previous failure."\n summary: "One or more placement groups are marked unclean on cluster {{ $labels.cluster }}"\n expr: "ceph_pool_metadata * on(cluster,pool_id,instance) group_left() (ceph_pg_total - ceph_pg_clean) > 0"\n for: "15m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.2"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPGsDamaged"\n annotations:\n description: "During data consistency checks (scrub), at least one PG has been flagged as being damaged or inconsistent. Check to see which PG is affected, and attempt a manual repair if necessary. To list problematic placement groups, use \'rados list-inconsistent-pg \'. To repair PGs use the \'ceph pg repair \' command."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-damaged"\n summary: "Placement group damaged, manual intervention needed on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=~\\"PG_DAMAGED|OSD_SCRUB_ERRORS\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.4"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGRecoveryAtRisk"\n annotations:\n description: "Data redundancy is at risk since one or more OSDs are at or above the \'full\' threshold. Add more capacity to the cluster, restore down/out OSDs, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-recovery-full"\n summary: "OSDs are too full for recovery on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"PG_RECOVERY_FULL\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.5"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGUnavailableBlockingIO"\n annotations:\n description: "Data availability is reduced, impacting the cluster\'s ability to service I/O. One or more placement groups (PGs) are in a state that blocks I/O."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-availability"\n summary: "PG is unavailable on cluster {{ $labels.cluster }}, blocking I/O"\n expr: "((ceph_health_detail{name=\\"PG_AVAILABILITY\\"} == 1) - scalar(ceph_health_detail{name=\\"OSD_DOWN\\"})) == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.3"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGBackfillAtRisk"\n annotations:\n description: "Data redundancy may be at risk due to lack of free space within the cluster. One or more OSDs have reached the \'backfillfull\' threshold. Add more capacity, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-backfill-full"\n summary: "Backfill operations are blocked due to lack of free space on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"PG_BACKFILL_FULL\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.6"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGNotScrubbed"\n annotations:\n description: "One or more PGs have not been scrubbed recently. Scrubs check metadata integrity, protecting against bit-rot. They check that metadata is consistent across data replicas. When PGs miss their scrub interval, it may indicate that the scrub window is too small, or PGs were not in a \'clean\' state during the scrub window. You can manually initiate a scrub with: ceph pg scrub "\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-not-scrubbed"\n summary: "Placement group(s) have not been scrubbed on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"PG_NOT_SCRUBBED\\"} == 1"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPGsHighPerOSD"\n annotations:\n description: "The number of placement groups per OSD is too high (exceeds the mon_max_pg_per_osd setting).\\n Check that the pg_autoscaler has not been disabled for any pools with \'ceph osd pool autoscale-status\', and that the profile selected is appropriate. You may also adjust the target_size_ratio of a pool to guide the autoscaler based on the expected relative size of the pool (\'ceph osd pool set cephfs.cephfs.meta target_size_ratio .1\') or set the pg_autoscaler mode to \'warn\' and adjust pg_num appropriately for one or more pools."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks/#too-many-pgs"\n summary: "Placement groups per OSD is too high on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"TOO_MANY_PGS\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPGNotDeepScrubbed"\n annotations:\n description: "One or more PGs have not been deep scrubbed recently. Deep scrubs protect against bit-rot. They compare data replicas to ensure consistency. When PGs miss their deep scrub interval, it may indicate that the window is too small or PGs were not in a \'clean\' state during the deep-scrub window."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-not-deep-scrubbed"\n summary: "Placement group(s) have not been deep scrubbed on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"PG_NOT_DEEP_SCRUBBED\\"} == 1"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "nodes"\n rules:\n - alert: "CephNodeRootFilesystemFull"\n annotations:\n description: "Root volume is dangerously full: {{ $value | humanize }}% free."\n summary: "Root filesystem is dangerously full"\n expr: "node_filesystem_avail_bytes{mountpoint=\\"/\\"} / node_filesystem_size_bytes{mountpoint=\\"/\\"} * 100 < 5"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.8.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephNodeNetworkPacketDrops"\n annotations:\n description: "Node {{ $labels.instance }} experiences packet drop > 0.5% or > 10 packets/s on interface {{ $labels.device }}."\n summary: "One or more NICs reports packet drops"\n expr: |\n (\n rate(node_network_receive_drop_total{device!="lo"}[1m]) +\n rate(node_network_transmit_drop_total{device!="lo"}[1m])\n ) / (\n rate(node_network_receive_packets_total{device!="lo"}[1m]) +\n rate(node_network_transmit_packets_total{device!="lo"}[1m])\n ) >= 0.0050000000000000001 and (\n rate(node_network_receive_drop_total{device!="lo"}[1m]) +\n rate(node_network_transmit_drop_total{device!="lo"}[1m])\n ) >= 10\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.8.2"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephNodeNetworkPacketErrors"\n annotations:\n description: "Node {{ $labels.instance }} experiences packet errors > 0.01% or > 10 packets/s on interface {{ $labels.device }}."\n summary: "One or more NICs reports packet errors on cluster {{ $labels.cluster }}"\n expr: |\n (\n rate(node_network_receive_errs_total{device!="lo"}[1m]) +\n rate(node_network_transmit_errs_total{device!="lo"}[1m])\n ) / (\n rate(node_network_receive_packets_total{device!="lo"}[1m]) +\n rate(node_network_transmit_packets_total{device!="lo"}[1m])\n ) >= 0.0001 or (\n rate(node_network_receive_errs_total{device!="lo"}[1m]) +\n rate(node_network_transmit_errs_total{device!="lo"}[1m])\n ) >= 10\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.8.3"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephNodeNetworkBondDegraded"\n annotations:\n description: "Bond {{ $labels.master }} is degraded on Node {{ $labels.instance }}."\n summary: "Degraded Bond on Node {{ $labels.instance }} on cluster {{ $labels.cluster }}"\n expr: |\n node_bonding_slaves - node_bonding_active != 0\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephNodeDiskspaceWarning"\n annotations:\n description: "Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will be full in less than 5 days based on the 48 hour trailing fill rate."\n summary: "Host filesystem free space is getting low on cluster {{ $labels.cluster }}"\n expr: "predict_linear(node_filesystem_free_bytes{device=~\\"/.*\\"}[2d], 3600 * 24 * 5) * on(cluster, instance) group_left(nodename) node_uname_info < 0"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.8.4"\n severity: "warning"\n type: "ceph_default"\n - alert: CephNodeInconsistentMTU\n expr: |\n node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)\n != on (cluster, device) group_left\n quantile by (cluster, device) (\n 0.5, node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)\n )\n labels:\n severity: warning\n type: ceph_default\n annotations:\n summary: "Node {{ $labels.instance }} has inconsistent MTU settings in cluster {{ $labels.cluster }}"\n description: "Network interface {{ $labels.device }} on node {{ $labels.instance }} has MTU {{ $value }} which differs from the cluster median."\n impact: |\n - May cause packet fragmentation or packet drops\n - Risk of degraded cluster communication and performance\n - Potential instability in services relying on consistent networking (e.g., Ceph, Kubernetes)\n fix: |\n - Check the MTU of interface `{{ $labels.device }}` on node `{{ $labels.instance }}`:\n ip link show {{ $labels.device }}\n\n - Find the median MTU value across the cluster by running this PromQL query in Prometheus:\n quantile by (cluster, device) (0.5, node_network_mtu_bytes * (node_network_up{device!="lo"} > 0))\n\n - Standardize MTU across all nodes to match the median (commonly 1500 or 9000):\n ip link set dev {{ $labels.device }} mtu \n\n - Make MTU setting persistent:\n - RHEL/CentOS: edit `/etc/sysconfig/network-scripts/ifcfg-`\n - Debian/Ubuntu: edit `/etc/netplan/*.yaml` and apply with `netplan apply`\n\n - Restart the affected interface or node if required.\n - name: "pools"\n rules:\n - alert: "CephPoolGrowthWarning"\n annotations:\n description: "Pool \'{{ $labels.name }}\' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours."\n summary: "Pool growth rate may soon exceed capacity on cluster {{ $labels.cluster }}"\n expr: "(predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(cluster,pool_id, instance) group_right() ceph_pool_metadata) >= 95"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.9.2"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPoolBackfillFull"\n annotations:\n description: "A pool is approaching the near full threshold, which will prevent recovery/backfill operations from completing. Consider adding more capacity."\n summary: "Free space in a pool is too low for recovery/backfill on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"POOL_BACKFILLFULL\\"} > 0"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPoolFull"\n annotations:\n description: "A pool has reached its MAX quota, or OSDs supporting the pool have reached the FULL threshold. Until this is resolved, writes to the pool will be blocked. Pool Breakdown (top 5) {{- range printf \\"topk(5, sort_desc(ceph_pool_percent_used{cluster=\'%s\'} * on(cluster,pool_id) group_right ceph_pool_metadata))\\" .Labels.cluster | query }} - {{ .Labels.name }} at {{ .Value }}% {{- end }} Increase the pool\'s quota, or add capacity to the cluster first then increase the pool\'s quota (e.g. ceph osd pool set quota max_bytes )"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pool-full"\n summary: "Pool is full - writes are blocked on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"POOL_FULL\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.9.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPoolNearFull"\n annotations:\n description: "A pool has exceeded the warning (percent full) threshold, or OSDs supporting the pool have reached the NEARFULL threshold. Writes may continue, but you are at risk of the pool going read-only if more capacity isn\'t made available. Determine the affected pool with \'ceph df detail\', looking at QUOTA BYTES and STORED. Increase the pool\'s quota, or add capacity to the cluster first then increase the pool\'s quota (e.g. ceph osd pool set quota max_bytes ). Also ensure that the balancer is active."\n summary: "One or more Ceph pools are nearly full on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"POOL_NEAR_FULL\\"} > 0"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "healthchecks"\n rules:\n - alert: "CephSlowOps"\n annotations:\n description: "{{ $value }} OSD requests are taking too long to process (osd_op_complaint_time exceeded)"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#slow-ops"\n summary: "OSD operations are slow to complete on cluster {{ $labels.cluster }}"\n expr: "ceph_healthcheck_slow_ops > 0"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephDaemonSlowOps"\n annotations:\n description: "{{ $labels.ceph_daemon }} operations are taking too long to process (complaint time exceeded)"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#slow-ops"\n summary: "{{ $labels.ceph_daemon }} operations are slow to complete on cluster {{ $labels.cluster }}"\n expr: "ceph_daemon_health_metrics{type=\\"SLOW_OPS\\"} > 0"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "cephadm"\n rules:\n - alert: "CephadmUpgradeFailed"\n annotations:\n description: "The cephadm cluster upgrade process has failed. The cluster remains in an undetermined state. Please review the cephadm logs, to understand the nature of the issue"\n summary: "Ceph version upgrade has failed on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"UPGRADE_EXCEPTION\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.11.2"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephadmDaemonFailed"\n annotations:\n description: "A daemon managed by cephadm is no longer active. Determine, which daemon is down with \'ceph health detail\'. you may start daemons with the \'ceph orch daemon start \'"\n summary: "A ceph daemon managed by cephadm is down on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"CEPHADM_FAILED_DAEMON\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.11.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephadmPaused"\n annotations:\n description: "Cluster management has been paused manually. This will prevent the orchestrator from service management and reconciliation. If this is not intentional, resume cephadm operations with \'ceph orch resume\'"\n documentation: "https://docs.ceph.com/en/latest/cephadm/operations#cephadm-paused"\n summary: "Orchestration tasks via cephadm are PAUSED on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"CEPHADM_PAUSED\\"} > 0"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "hardware"\n rules:\n - alert: "HardwareStorageError"\n annotations:\n description: "Some storage devices are in error. Check `ceph health detail`."\n summary: "Storage devices error(s) detected on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"HARDWARE_STORAGE\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwareMemoryError"\n annotations:\n description: "DIMM error(s) detected. Check `ceph health detail`."\n summary: "DIMM error(s) detected on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"HARDWARE_MEMORY\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.2"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwareProcessorError"\n annotations:\n description: "Processor error(s) detected. Check `ceph health detail`."\n summary: "Processor error(s) detected on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"HARDWARE_PROCESSOR\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.3"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwareNetworkError"\n annotations:\n description: "Network error(s) detected. Check `ceph health detail`."\n summary: "Network error(s) detected on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"HARDWARE_NETWORK\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.4"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwarePowerError"\n annotations:\n description: "Power supply error(s) detected. Check `ceph health detail`."\n summary: "Power supply error(s) detected on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"HARDWARE_POWER\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.5"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwareFanError"\n annotations:\n description: "Fan error(s) detected. Check `ceph health detail`."\n summary: "Fan error(s) detected on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"HARDWARE_FANS\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.6"\n severity: "critical"\n type: "ceph_default"\n - name: "PrometheusServer"\n rules:\n - alert: "PrometheusJobMissing"\n annotations:\n description: "The prometheus job that scrapes from Ceph is no longer defined, this will effectively mean you\'ll have no metrics or alerts for the cluster. Please review the job definitions in the prometheus.yml file of the prometheus instance."\n summary: "The scrape job for Ceph is missing from Prometheus"\n expr: "absent(up{job=\\"ceph\\"})"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.12.1"\n severity: "critical"\n type: "ceph_default"\n - name: "rados"\n rules:\n - alert: "CephObjectMissing"\n annotations:\n description: "The latest version of a RADOS object can not be found, even though all OSDs are up. I/O requests for this object from clients will block (hang). Resolving this issue may require the object to be rolled back to a prior version manually, and manually verified."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#object-unfound"\n summary: "Object(s) marked UNFOUND on cluster {{ $labels.cluster }}"\n expr: "(ceph_health_detail{name=\\"OBJECT_UNFOUND\\"} == 1) * on() group_right(cluster) (count(ceph_osd_up == 1) by (cluster) == bool count(ceph_osd_metadata) by(cluster)) == 1"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.1"\n severity: "critical"\n type: "ceph_default"\n - name: "generic"\n rules:\n - alert: "CephDaemonCrash"\n annotations:\n description: "One or more daemons have crashed recently, and need to be acknowledged. This notification ensures that software crashes do not go unseen. To acknowledge a crash, use the \'ceph crash archive \' command."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks/#recent-crash"\n summary: "One or more Ceph daemons have crashed, and are pending acknowledgement on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"RECENT_CRASH\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.1.2"\n severity: "critical"\n type: "ceph_default"\n - name: "rbdmirror"\n rules:\n - alert: "CephRBDMirrorImagesPerDaemonHigh"\n annotations:\n description: "Number of image replications per daemon is not supposed to go beyond threshold 100"\n summary: "Number of image replications are now above 100 on cluster {{ $labels.cluster }}"\n expr: "sum by (cluster, ceph_daemon, namespace) (ceph_rbd_mirror_snapshot_image_snapshots) > 100"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.2"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephRBDMirrorImagesNotInSync"\n annotations:\n description: "Both local and remote RBD mirror images should be in sync."\n summary: "Some of the RBD mirror images are not in sync with the remote counter parts on cluster {{ $labels.cluster }}"\n expr: "sum by (cluster, ceph_daemon, image, namespace, pool) (topk by (cluster, ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_local_timestamp) - topk by (cluster, ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_remote_timestamp)) != 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.3"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephRBDMirrorImagesNotInSyncVeryHigh"\n annotations:\n description: "More than 10% of the images have synchronization problems."\n summary: "Number of unsynchronized images are very high on cluster {{ $labels.cluster }}"\n expr: "count by (ceph_daemon, cluster) ((topk by (cluster, ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_local_timestamp) - topk by (cluster, ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_remote_timestamp)) != 0) > (sum by (ceph_daemon, cluster) (ceph_rbd_mirror_snapshot_snapshots)*.1)"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.4"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephRBDMirrorImageTransferBandwidthHigh"\n annotations:\n description: "Detected a heavy increase in bandwidth for rbd replications (over 80%) in the last 30 min. This might not be a problem, but it is good to review the number of images being replicated simultaneously"\n summary: "The replication network usage on cluster {{ $labels.cluster }} has been increased over 80% in the last 30 minutes. Review the number of images being replicated. This alert will be cleaned automatically after 30 minutes"\n expr: "rate(ceph_rbd_mirror_journal_replay_bytes[30m]) > 0.80"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.5"\n severity: "warning"\n type: "ceph_default"\n - name: "nvmeof"\n rules:\n - alert: "NVMeoFSubsystemNamespaceLimit"\n annotations:\n description: "Subsystems have a max namespace limit defined at creation time. This alert means that no more namespaces can be added to {{ $labels.nqn }}"\n summary: "{{ $labels.nqn }} subsystem has reached its maximum number of namespaces on cluster {{ $labels.cluster }}"\n expr: "(count by(nqn, cluster, instance) (ceph_nvmeof_subsystem_namespace_metadata)) >= on(nqn, instance) group_right(cluster) ceph_nvmeof_subsystem_namespace_limit"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFMultipleNamespacesOfRBDImage"\n annotations:\n description: "Each NVMeoF namespace must have a unique RBD pool and image, across all different gateway groups."\n summary: "RBD image {{ $labels.pool_name }}/{{ $labels.rbd_name }} cannot be reused for multiple NVMeoF namespace "\n expr: "count by(pool_name, rbd_name) (count by(bdev_name, pool_name, rbd_name) (ceph_nvmeof_bdev_metadata and on (bdev_name) ceph_nvmeof_subsystem_namespace_metadata)) > 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFTooManyGateways"\n annotations:\n description: "You may create many gateways, but 32 is the tested limit"\n summary: "Max supported gateways exceeded on cluster {{ $labels.cluster }}"\n expr: "count(ceph_nvmeof_gateway_info) by (cluster) > 32.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFMaxGatewayGroupSize"\n annotations:\n description: "You may create many gateways in a gateway group, but 8 is the tested limit"\n summary: "Max gateways within a gateway group ({{ $labels.group }}) exceeded on cluster {{ $labels.cluster }}"\n expr: "count(ceph_nvmeof_gateway_info) by (cluster,group) > 8.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFMaxGatewayGroups"\n annotations:\n description: "You may create many gateway groups, but 4 is the tested limit"\n summary: "Max gateway groups exceeded on cluster {{ $labels.cluster }}"\n expr: "count(count by (group, cluster) (ceph_nvmeof_gateway_info)) by (cluster) > 4.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFSingleGateway"\n annotations:\n description: "Although a single member gateway group is valid, it should only be used for test purposes"\n summary: "The gateway group {{ $labels.group }} consists of a single gateway - HA is not possible on cluster {{ $labels.cluster }}"\n expr: "count(ceph_nvmeof_gateway_info) by(cluster,group) == 1"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighGatewayCPU"\n annotations:\n description: "Typically, high CPU may indicate degraded performance. Consider increasing the number of reactor cores"\n summary: "CPU used by {{ $labels.instance }} NVMe-oF Gateway is high on cluster {{ $labels.cluster }}"\n expr: "label_replace(avg by(instance, cluster) (rate(ceph_nvmeof_reactor_seconds_total{mode=\\"busy\\"}[1m])),\\"instance\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\") > 80.00"\n for: "10m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFGatewayOpenSecurity"\n annotations:\n description: "It is good practice to ensure subsystems use host security to reduce the risk of unexpected data loss"\n summary: "Subsystem {{ $labels.nqn }} has been defined without host level security on cluster {{ $labels.cluster }}"\n expr: "ceph_nvmeof_subsystem_metadata{allow_any_host=\\"yes\\"}"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFTooManySubsystems"\n annotations:\n description: "NVMeoF gateway {{ $labels.gateway_host }} has reached or exceeded the supported maximum of 128 subsystems. Current count: {{ $value }}."\n summary: "The number of subsystems defined to the NVMeoF gateway reached or exceeded the supported values on cluster {{ $labels.cluster }}"\n expr: "count by(gateway_host, cluster) (label_replace(ceph_nvmeof_subsystem_metadata,\\"gateway_host\\",\\"$1\\",\\"instance\\",\\"(.*?)(?::.*)?\\")) >= 128.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFTooManyNamespaces"\n annotations:\n description: "NVMeoF gateway {{ $labels.gateway_host }} has reached or exceeded the supported maximum of 4096 namespaces. Current count: {{ $value }}."\n summary: "The number of namespaces defined to the NVMeoF gateway reached or exceeded supported values on cluster {{ $labels.cluster }}"\n expr: "sum by(gateway_host, cluster) (label_replace(ceph_nvmeof_subsystem_namespace_count,\\"gateway_host\\",\\"$1\\",\\"instance\\",\\"(.*?)(?::.*)?\\")) >= 4096.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFVersionMismatch"\n annotations:\n description: "This may indicate an issue with deployment. Check cephadm logs"\n summary: "Too many different NVMe-oF gateway releases active on cluster {{ $labels.cluster }}"\n expr: "count(count(ceph_nvmeof_gateway_info) by (cluster, version)) by (cluster) > 1"\n for: "1h"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighClientCount"\n annotations:\n description: "The supported limit for clients connecting to a subsystem is 128"\n summary: "The number of clients connected to {{ $labels.nqn }} is too high on cluster {{ $labels.cluster }}"\n expr: "ceph_nvmeof_subsystem_host_count > 128.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFMissingListener"\n annotations:\n description: "For every subsystem, each gateway should have a listener to balance traffic between gateways."\n summary: "No listener added for {{ $labels.instance }} NVMe-oF Gateway to {{ $labels.nqn }} subsystem"\n expr: "ceph_nvmeof_subsystem_listener_count == 0 and on(nqn) sum(ceph_nvmeof_subsystem_listener_count) by (nqn) > 0"\n for: "10m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFZeroListenerSubsystem"\n annotations:\n description: "NVMeoF gateway configuration incomplete; one of the subsystems have zero listeners."\n summary: "No listeners added to {{ $labels.nqn }} subsystem"\n expr: "sum(ceph_nvmeof_subsystem_listener_count) by (nqn) == 0"\n for: "10m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighHostCPU"\n annotations:\n description: "High CPU on a gateway host can lead to CPU contention and performance degradation"\n summary: "The CPU is high ({{ $value }}%) on NVMeoF Gateway host ({{ $labels.host }}) on cluster {{ $labels.cluster }}"\n expr: "100-((100*(avg by(cluster,host) (label_replace(rate(node_cpu_seconds_total{mode=\\"idle\\"}[5m]),\\"host\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\")) * on(cluster, host) group_right label_replace(ceph_nvmeof_gateway_info,\\"host\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\")))) >= 80.00"\n for: "10m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFInterfaceDown"\n annotations:\n description: "A NIC used by one or more subsystems is in a down state"\n summary: "Network interface {{ $labels.device }} is down on cluster {{ $labels.cluster }}"\n expr: "ceph_nvmeof_subsystem_listener_iface_info{operstate=\\"down\\"}"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.14.1"\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFInterfaceDuplex"\n annotations:\n description: "Until this is resolved, performance from the gateway will be degraded"\n summary: "Network interface {{ $labels.device }} is not running in full duplex mode on cluster {{ $labels.cluster }}"\n expr: "ceph_nvmeof_subsystem_listener_iface_info{duplex!=\\"full\\"}"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighReadLatency"\n annotations:\n description: "High latencies may indicate a constraint within the cluster e.g. CPU, network. Please investigate"\n summary: "The average read latency over the last 5 mins has reached 10 ms or more on {{ $labels.gateway }}"\n expr: "label_replace((avg by(instance) ((rate(ceph_nvmeof_bdev_read_seconds_total[1m]) / rate(ceph_nvmeof_bdev_reads_completed_total[1m])))),\\"gateway\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\") > 0.01"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighWriteLatency"\n annotations:\n description: "High latencies may indicate a constraint within the cluster e.g. CPU, network. Please investigate"\n summary: "The average write latency over the last 5 mins has reached 20 ms or more on {{ $labels.gateway }}"\n expr: "label_replace((avg by(instance) ((rate(ceph_nvmeof_bdev_write_seconds_total[5m]) / rate(ceph_nvmeof_bdev_writes_completed_total[5m])))),\\"gateway\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\") > 0.02"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHostKeepAliveTimeout"\n annotations:\n description: "Host was disconnected due to host keep alive timeout"\n summary: "Host ({{ $labels.host_nqn }}) was disconnected {{ $value }} times from subsystem ({{ $labels.nqn }}) in last 24 hours"\n expr: "ceil(changes(ceph_nvmeof_host_keepalive_timeout[24h:]) / 2) > 0"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n', '/etc/prometheus/alerting/custom_alerts.yml': ''}, 'retention_time': '15d', 'retention_size': '0', 'ip_to_bind_to': '', 'use_url_prefix': False}} 2026-02-14 01:36:54,328 7fa365c09e00 DEBUG Determined image: 'quay.io/prometheus/prometheus:v2.51.0' 2026-02-14 01:36:54,342 7fa365c09e00 INFO Reconfig daemon prometheus.trial014 ... 2026-02-14 01:36:54,555 7fa365c09e00 DEBUG firewalld.service is not enabled 2026-02-14 01:36:54,555 7fa365c09e00 DEBUG Not possible to open ports <[9095]>. firewalld.service is not available 2026-02-14 01:36:56,285 7fa0a3bbce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ls'] 2026-02-14 01:36:59,652 7f1e6da57e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'inspect-image'] 2026-02-14 01:36:59,688 7f1e6da57e00 DEBUG /usr/bin/podman: stdout edc95c9460f10a90f1e921f9347538ed9be0b1c7aee54201d89be4c7dacb23c7,[quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9] 2026-02-14 01:36:59,816 7f1e6da57e00 DEBUG ceph: stdout ceph version 20.2.0-667-geae49298 (eae49298735c216a37f3957204b0fe058b58e376) tentacle (stable) 2026-02-14 01:36:59,953 7f22bb428e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:36:59,969 7f22bb428e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'mon.trial014', 'image': '', 'deploy_arguments': [], 'params': {'allow_ptrace': True}, 'meta': {'service_name': 'mon', 'ports': [], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n[mon.trial014]\npublic network = 10.20.192.0/20\n', 'keyring': '[mon.]\n\tkey = AQDz0I9p5ZHrBRAAqsLJRevJFlCa8pfULLdJ7Q==\n\tcaps mon = "allow *"\n', 'files': {'config': '[mon.trial014]\npublic network = 10.20.192.0/20\n'}}} 2026-02-14 01:36:59,969 7f22bb428e00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9' 2026-02-14 01:36:59,982 7f22bb428e00 INFO Redeploy daemon mon.trial014 ... 2026-02-14 01:37:00,781 7f22bb428e00 DEBUG firewalld.service is not enabled 2026-02-14 01:37:00,782 7f22bb428e00 DEBUG Not possible to enable service . firewalld.service is not available 2026-02-14 01:38:42,725 7efdf4ec5e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ls'] 2026-02-14 01:38:43,890 7fb01821ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:38:44,144 7feb02647e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'list-networks'] 2026-02-14 01:38:48,257 7fdcad024e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:38:48,276 7fdcad024e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'mon.trial014', 'image': '', 'deploy_arguments': [], 'params': {'reconfig': True, 'allow_ptrace': True}, 'meta': {'service_name': 'mon', 'ports': [], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n[mon.trial014]\npublic network = 10.20.192.0/20\n', 'keyring': '[mon.]\n\tkey = AQDz0I9p5ZHrBRAAqsLJRevJFlCa8pfULLdJ7Q==\n\tcaps mon = "allow *"\n', 'files': {'config': '[mon.trial014]\npublic network = 10.20.192.0/20\n'}}} 2026-02-14 01:38:48,276 7fdcad024e00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9' 2026-02-14 01:38:48,290 7fdcad024e00 INFO Reconfig daemon mon.trial014 ... 2026-02-14 01:38:48,386 7fdcad024e00 DEBUG firewalld.service is not enabled 2026-02-14 01:38:48,386 7fdcad024e00 DEBUG Not possible to enable service . firewalld.service is not available 2026-02-14 01:38:48,521 7f2909a1ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:38:48,537 7f2909a1ce00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'mgr.trial014.uucrmp', 'image': '', 'deploy_arguments': [], 'params': {'tcp_ports': [8443, 9283, 8765], 'reconfig': True, 'allow_ptrace': True}, 'meta': {'service_name': 'mgr', 'ports': [8443, 9283, 8765], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[mgr.trial014.uucrmp]\nkey = AQDz0I9pYGlEFhAAOnF6ryAzuu/H9SeCFjrELQ==\n'}} 2026-02-14 01:38:48,537 7f2909a1ce00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9' 2026-02-14 01:38:48,550 7f2909a1ce00 INFO Reconfig daemon mgr.trial014.uucrmp ... 2026-02-14 01:38:48,635 7f2909a1ce00 DEBUG firewalld.service is not enabled 2026-02-14 01:38:48,635 7f2909a1ce00 DEBUG Not possible to enable service . firewalld.service is not available 2026-02-14 01:38:48,647 7f2909a1ce00 DEBUG firewalld.service is not enabled 2026-02-14 01:38:48,647 7f2909a1ce00 DEBUG Not possible to open ports <[8443, 9283, 8765]>. firewalld.service is not available 2026-02-14 01:38:48,793 7fc4394d7e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:38:48,809 7fc4394d7e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'ceph-exporter.trial014', 'image': '', 'deploy_arguments': [], 'params': {'tcp_ports': [9926], 'reconfig': True, 'allow_ptrace': True}, 'meta': {'service_name': 'ceph-exporter', 'ports': [9926], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[client.ceph-exporter.trial014]\nkey = AQAL0Y9pSQnMEBAAd1kOyDHxNKSk3p930enz8A==\n', 'prio-limit': '5', 'stats-period': '5'}} 2026-02-14 01:38:48,809 7fc4394d7e00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4' 2026-02-14 01:38:48,822 7fc4394d7e00 INFO Reconfig daemon ceph-exporter.trial014 ... 2026-02-14 01:38:48,972 7fc4394d7e00 INFO Writing ceph-exporter config... 2026-02-14 01:38:48,986 7fc4394d7e00 DEBUG firewalld.service is not enabled 2026-02-14 01:38:48,986 7fc4394d7e00 DEBUG Not possible to open ports <[9926]>. firewalld.service is not available 2026-02-14 01:38:49,122 7fbd543c6e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:38:49,138 7fbd543c6e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'crash.trial014', 'image': '', 'deploy_arguments': [], 'params': {'reconfig': True, 'allow_ptrace': True}, 'meta': {'service_name': 'crash', 'ports': [], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[client.crash.trial014]\nkey = AQAL0Y9pPH3KOhAANHuSa4hyjyB/nHdHqJVEsg==\n'}} 2026-02-14 01:38:49,138 7fbd543c6e00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4' 2026-02-14 01:38:49,150 7fbd543c6e00 INFO Reconfig daemon crash.trial014 ... 2026-02-14 01:38:49,358 7f27cc260e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:38:49,374 7f27cc260e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'osd.0', 'image': '', 'deploy_arguments': [], 'params': {'osd_fsid': '1d35940b-bfca-4f46-8c96-9c8c06e3c945', 'reconfig': True, 'allow_ptrace': True}, 'meta': {'service_name': 'osd', 'ports': [], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[osd.0]\n\tkey = AQAr0Y9pqEb8FxAAnMomsBWtNxWw7AGx89cXug==\n\tcaps mgr = "allow profile osd"\n\tcaps mon = "allow profile osd"\n\tcaps osd = "allow *"\n'}} 2026-02-14 01:38:49,375 7f27cc260e00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4' 2026-02-14 01:38:49,387 7f27cc260e00 INFO Reconfig daemon osd.0 ... 2026-02-14 01:38:49,517 7f27cc260e00 DEBUG firewalld.service is not enabled 2026-02-14 01:38:49,517 7f27cc260e00 DEBUG Not possible to enable service . firewalld.service is not available 2026-02-14 01:38:49,651 7f55c5e62e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:38:49,668 7f55c5e62e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'osd.1', 'image': '', 'deploy_arguments': [], 'params': {'osd_fsid': '89abfd8b-9842-4712-b460-553af014b17c', 'reconfig': True, 'allow_ptrace': True}, 'meta': {'service_name': 'osd', 'ports': [], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[osd.1]\n\tkey = AQA20Y9pUXGvIBAARoK0cvUPLYkYVRtXxjUDZw==\n\tcaps mgr = "allow profile osd"\n\tcaps mon = "allow profile osd"\n\tcaps osd = "allow *"\n'}} 2026-02-14 01:38:49,668 7f55c5e62e00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4' 2026-02-14 01:38:49,680 7f55c5e62e00 INFO Reconfig daemon osd.1 ... 2026-02-14 01:38:49,794 7f55c5e62e00 DEBUG firewalld.service is not enabled 2026-02-14 01:38:49,794 7f55c5e62e00 DEBUG Not possible to enable service . firewalld.service is not available 2026-02-14 01:38:49,929 7fa02c413e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:38:49,945 7fa02c413e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'osd.2', 'image': '', 'deploy_arguments': [], 'params': {'osd_fsid': 'ce7d3155-0e14-4989-baf6-12e50590fb75', 'reconfig': True, 'allow_ptrace': True}, 'meta': {'service_name': 'osd', 'ports': [], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[osd.2]\n\tkey = AQBB0Y9pRK6AMhAA3PlNX4pWXFXkKXOn5xJkaA==\n\tcaps mgr = "allow profile osd"\n\tcaps mon = "allow profile osd"\n\tcaps osd = "allow *"\n'}} 2026-02-14 01:38:49,945 7fa02c413e00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4' 2026-02-14 01:38:49,958 7fa02c413e00 INFO Reconfig daemon osd.2 ... 2026-02-14 01:38:50,065 7fa02c413e00 DEBUG firewalld.service is not enabled 2026-02-14 01:38:50,065 7fa02c413e00 DEBUG Not possible to enable service . firewalld.service is not available 2026-02-14 01:38:50,199 7f5bf6c47e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:38:50,215 7f5bf6c47e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'mds.cephfs.trial014.ubresr', 'image': '', 'deploy_arguments': [], 'params': {'reconfig': True, 'allow_ptrace': True}, 'meta': {'service_name': 'mds.cephfs', 'ports': [], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[mds.cephfs.trial014.ubresr]\nkey = AQCB0Y9paSGPGBAAe4qWrzsf3JPgerwk3WRkCA==\n'}} 2026-02-14 01:38:50,215 7f5bf6c47e00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4' 2026-02-14 01:38:50,229 7f5bf6c47e00 INFO Reconfig daemon mds.cephfs.trial014.ubresr ... 2026-02-14 01:38:50,306 7f5bf6c47e00 DEBUG firewalld.service is not enabled 2026-02-14 01:38:50,306 7f5bf6c47e00 DEBUG Not possible to enable service . firewalld.service is not available 2026-02-14 01:38:50,439 7f035aa8ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:38:50,454 7f035aa8ae00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'mds.cephfs.trial014.kwvbap', 'image': '', 'deploy_arguments': [], 'params': {'reconfig': True, 'allow_ptrace': True}, 'meta': {'service_name': 'mds.cephfs', 'ports': [], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[mds.cephfs.trial014.kwvbap]\nkey = AQCC0Y9pGiBGMxAA7p0e98mGukEB03+oiEk3QQ==\n'}} 2026-02-14 01:38:50,454 7f035aa8ae00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4' 2026-02-14 01:38:50,467 7f035aa8ae00 INFO Reconfig daemon mds.cephfs.trial014.kwvbap ... 2026-02-14 01:38:50,566 7f035aa8ae00 DEBUG firewalld.service is not enabled 2026-02-14 01:38:50,566 7f035aa8ae00 DEBUG Not possible to enable service . firewalld.service is not available 2026-02-14 01:38:53,090 7f9e0d282e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'inspect-image'] 2026-02-14 01:38:53,124 7f9e0d282e00 DEBUG /usr/bin/podman: stdout edc95c9460f10a90f1e921f9347538ed9be0b1c7aee54201d89be4c7dacb23c7,[quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9] 2026-02-14 01:38:53,243 7f9e0d282e00 DEBUG ceph: stdout ceph version 20.2.0-667-geae49298 (eae49298735c216a37f3957204b0fe058b58e376) tentacle (stable) 2026-02-14 01:38:53,377 7f12dc452e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:38:53,394 7f12dc452e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'crash.trial014', 'image': '', 'deploy_arguments': [], 'params': {'allow_ptrace': True}, 'meta': {'service_name': 'crash', 'ports': [], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[client.crash.trial014]\nkey = AQAL0Y9pPH3KOhAANHuSa4hyjyB/nHdHqJVEsg==\n'}} 2026-02-14 01:38:53,394 7f12dc452e00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9' 2026-02-14 01:38:53,407 7f12dc452e00 INFO Redeploy daemon crash.trial014 ... 2026-02-14 01:38:55,261 7f1977e3ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 01:38:56,347 7f5b62087e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:38:56,561 7fab855ebe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'list-networks'] 2026-02-14 01:38:56,718 7fe464829e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 01:38:56,735 7fe464829e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:38:56,736 7fe464829e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:38:57,472 7f84b22cce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'inspect-image'] 2026-02-14 01:38:57,507 7f84b22cce00 DEBUG /usr/bin/podman: stdout edc95c9460f10a90f1e921f9347538ed9be0b1c7aee54201d89be4c7dacb23c7,[quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9] 2026-02-14 01:38:57,630 7f84b22cce00 DEBUG ceph: stdout ceph version 20.2.0-667-geae49298 (eae49298735c216a37f3957204b0fe058b58e376) tentacle (stable) 2026-02-14 01:38:57,761 7fe998c04e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:38:57,776 7fe998c04e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'osd.0', 'image': '', 'deploy_arguments': [], 'params': {'osd_fsid': '1d35940b-bfca-4f46-8c96-9c8c06e3c945', 'allow_ptrace': True}, 'meta': {'service_name': 'osd', 'ports': [], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[osd.0]\n\tkey = AQAr0Y9pqEb8FxAAnMomsBWtNxWw7AGx89cXug==\n\tcaps mgr = "allow profile osd"\n\tcaps mon = "allow profile osd"\n\tcaps osd = "allow *"\n'}} 2026-02-14 01:38:57,776 7fe998c04e00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9' 2026-02-14 01:38:57,790 7fe998c04e00 INFO Redeploy daemon osd.0 ... 2026-02-14 01:38:58,143 7fe998c04e00 DEBUG sysctl: stdout 1048576 2026-02-14 01:38:58,145 7fe998c04e00 DEBUG sysctl: stdout 4194304 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ... 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/50-coredump.conf ... 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/50-default.conf ... 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/50-libkcapi-optmem_max.conf ... 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/50-pid-max.conf ... 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/50-redhat.conf ... 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout * Applying /etc/sysctl.d/90-ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-osd.conf ... 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/90-ceph-osd.conf ... 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout * Applying /etc/sysctl.d/99-sysctl.conf ... 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout * Applying /etc/sysctl.conf ... 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout kernel.yama.ptrace_scope = 0 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout kernel.core_pipe_limit = 16 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout fs.suid_dumpable = 2 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout kernel.sysrq = 16 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout kernel.core_uses_pid = 1 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout net.ipv4.conf.default.rp_filter = 2 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout net.ipv4.conf.enp1s0f0.rp_filter = 2 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout net.ipv4.conf.enp1s0f1.rp_filter = 2 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout net.ipv4.conf.enp7s0f4u2u2c2.rp_filter = 2 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout net.ipv4.conf.lo.rp_filter = 2 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout net.ipv4.conf.default.accept_source_route = 0 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout net.ipv4.conf.enp1s0f0.accept_source_route = 0 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout net.ipv4.conf.enp1s0f1.accept_source_route = 0 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout net.ipv4.conf.enp7s0f4u2u2c2.accept_source_route = 0 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout net.ipv4.conf.lo.accept_source_route = 0 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout net.ipv4.conf.default.promote_secondaries = 1 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout net.ipv4.conf.enp1s0f0.promote_secondaries = 1 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout net.ipv4.conf.enp1s0f1.promote_secondaries = 1 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout net.ipv4.conf.enp7s0f4u2u2c2.promote_secondaries = 1 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout net.ipv4.conf.lo.promote_secondaries = 1 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout net.ipv4.ping_group_range = 0 2147483647 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout net.core.default_qdisc = fq_codel 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout fs.protected_hardlinks = 1 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout fs.protected_symlinks = 1 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout fs.protected_regular = 1 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout fs.protected_fifos = 1 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout net.core.optmem_max = 81920 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout kernel.pid_max = 4194304 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout kernel.kptr_restrict = 1 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout net.ipv4.conf.default.rp_filter = 1 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout net.ipv4.conf.enp1s0f0.rp_filter = 1 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout net.ipv4.conf.enp1s0f1.rp_filter = 1 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout net.ipv4.conf.enp7s0f4u2u2c2.rp_filter = 1 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout net.ipv4.conf.lo.rp_filter = 1 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout fs.aio-max-nr = 2097152 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout fs.aio-max-nr = 1048576 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout kernel.pid_max = 4194304 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-14 01:38:58,148 7fe998c04e00 DEBUG sysctl: stdout kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-14 01:39:00,949 7fe998c04e00 DEBUG firewalld.service is not enabled 2026-02-14 01:39:00,949 7fe998c04e00 DEBUG Not possible to enable service . firewalld.service is not available 2026-02-14 01:39:01,085 7fabaf283e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 01:39:02,206 7f53e6a2ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:39:02,422 7f67e54c1e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'list-networks'] 2026-02-14 01:39:02,574 7f5b3fbc5e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 01:39:02,589 7f5b3fbc5e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:39:02,589 7f5b3fbc5e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:39:03,323 7f8721676e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'inspect-image'] 2026-02-14 01:39:03,358 7f8721676e00 DEBUG /usr/bin/podman: stdout edc95c9460f10a90f1e921f9347538ed9be0b1c7aee54201d89be4c7dacb23c7,[quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9] 2026-02-14 01:39:03,472 7f8721676e00 DEBUG ceph: stdout ceph version 20.2.0-667-geae49298 (eae49298735c216a37f3957204b0fe058b58e376) tentacle (stable) 2026-02-14 01:39:03,601 7fedd7a12e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:39:03,617 7fedd7a12e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'osd.1', 'image': '', 'deploy_arguments': [], 'params': {'osd_fsid': '89abfd8b-9842-4712-b460-553af014b17c', 'allow_ptrace': True}, 'meta': {'service_name': 'osd', 'ports': [], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[osd.1]\n\tkey = AQA20Y9pUXGvIBAARoK0cvUPLYkYVRtXxjUDZw==\n\tcaps mgr = "allow profile osd"\n\tcaps mon = "allow profile osd"\n\tcaps osd = "allow *"\n'}} 2026-02-14 01:39:03,617 7fedd7a12e00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9' 2026-02-14 01:39:03,632 7fedd7a12e00 INFO Redeploy daemon osd.1 ... 2026-02-14 01:39:03,972 7fedd7a12e00 DEBUG sysctl: stdout 1048576 2026-02-14 01:39:03,974 7fedd7a12e00 DEBUG sysctl: stdout 4194304 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ... 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/50-coredump.conf ... 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/50-default.conf ... 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/50-libkcapi-optmem_max.conf ... 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/50-pid-max.conf ... 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/50-redhat.conf ... 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout * Applying /etc/sysctl.d/90-ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-osd.conf ... 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/90-ceph-osd.conf ... 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout * Applying /etc/sysctl.d/99-sysctl.conf ... 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout * Applying /etc/sysctl.conf ... 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout kernel.yama.ptrace_scope = 0 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout kernel.core_pipe_limit = 16 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout fs.suid_dumpable = 2 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout kernel.sysrq = 16 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout kernel.core_uses_pid = 1 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout net.ipv4.conf.default.rp_filter = 2 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout net.ipv4.conf.enp1s0f0.rp_filter = 2 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout net.ipv4.conf.enp1s0f1.rp_filter = 2 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout net.ipv4.conf.enp7s0f4u2u2c2.rp_filter = 2 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout net.ipv4.conf.lo.rp_filter = 2 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout net.ipv4.conf.default.accept_source_route = 0 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout net.ipv4.conf.enp1s0f0.accept_source_route = 0 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout net.ipv4.conf.enp1s0f1.accept_source_route = 0 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout net.ipv4.conf.enp7s0f4u2u2c2.accept_source_route = 0 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout net.ipv4.conf.lo.accept_source_route = 0 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout net.ipv4.conf.default.promote_secondaries = 1 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout net.ipv4.conf.enp1s0f0.promote_secondaries = 1 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout net.ipv4.conf.enp1s0f1.promote_secondaries = 1 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout net.ipv4.conf.enp7s0f4u2u2c2.promote_secondaries = 1 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout net.ipv4.conf.lo.promote_secondaries = 1 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout net.ipv4.ping_group_range = 0 2147483647 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout net.core.default_qdisc = fq_codel 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout fs.protected_hardlinks = 1 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout fs.protected_symlinks = 1 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout fs.protected_regular = 1 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout fs.protected_fifos = 1 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout net.core.optmem_max = 81920 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout kernel.pid_max = 4194304 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout kernel.kptr_restrict = 1 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout net.ipv4.conf.default.rp_filter = 1 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout net.ipv4.conf.enp1s0f0.rp_filter = 1 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout net.ipv4.conf.enp1s0f1.rp_filter = 1 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout net.ipv4.conf.enp7s0f4u2u2c2.rp_filter = 1 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout net.ipv4.conf.lo.rp_filter = 1 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout fs.aio-max-nr = 2097152 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout fs.aio-max-nr = 1048576 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout kernel.pid_max = 4194304 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-14 01:39:03,978 7fedd7a12e00 DEBUG sysctl: stdout kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-14 01:39:06,333 7fedd7a12e00 DEBUG firewalld.service is not enabled 2026-02-14 01:39:06,333 7fedd7a12e00 DEBUG Not possible to enable service . firewalld.service is not available 2026-02-14 01:39:06,473 7f40d9828e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 01:39:07,599 7f035483fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:39:07,818 7f1bf2631e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'list-networks'] 2026-02-14 01:39:07,975 7f8a8c4c1e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 01:39:07,991 7f8a8c4c1e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:39:07,991 7f8a8c4c1e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:39:23,735 7f6f23e57e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'inspect-image'] 2026-02-14 01:39:23,771 7f6f23e57e00 DEBUG /usr/bin/podman: stdout edc95c9460f10a90f1e921f9347538ed9be0b1c7aee54201d89be4c7dacb23c7,[quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9] 2026-02-14 01:39:23,893 7f6f23e57e00 DEBUG ceph: stdout ceph version 20.2.0-667-geae49298 (eae49298735c216a37f3957204b0fe058b58e376) tentacle (stable) 2026-02-14 01:39:24,026 7faf01405e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:39:24,043 7faf01405e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'osd.2', 'image': '', 'deploy_arguments': [], 'params': {'osd_fsid': 'ce7d3155-0e14-4989-baf6-12e50590fb75', 'allow_ptrace': True}, 'meta': {'service_name': 'osd', 'ports': [], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[osd.2]\n\tkey = AQBB0Y9pRK6AMhAA3PlNX4pWXFXkKXOn5xJkaA==\n\tcaps mgr = "allow profile osd"\n\tcaps mon = "allow profile osd"\n\tcaps osd = "allow *"\n'}} 2026-02-14 01:39:24,043 7faf01405e00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9' 2026-02-14 01:39:24,057 7faf01405e00 INFO Redeploy daemon osd.2 ... 2026-02-14 01:39:24,393 7faf01405e00 DEBUG sysctl: stdout 1048576 2026-02-14 01:39:24,395 7faf01405e00 DEBUG sysctl: stdout 4194304 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ... 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/50-coredump.conf ... 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/50-default.conf ... 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/50-libkcapi-optmem_max.conf ... 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/50-pid-max.conf ... 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/50-redhat.conf ... 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout * Applying /etc/sysctl.d/90-ceph-29ce0a10-0945-11f1-b2be-d404e6e7d460-osd.conf ... 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout * Applying /usr/lib/sysctl.d/90-ceph-osd.conf ... 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout * Applying /etc/sysctl.d/99-sysctl.conf ... 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout * Applying /etc/sysctl.conf ... 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout kernel.yama.ptrace_scope = 0 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout kernel.core_pipe_limit = 16 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout fs.suid_dumpable = 2 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout kernel.sysrq = 16 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout kernel.core_uses_pid = 1 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout net.ipv4.conf.default.rp_filter = 2 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout net.ipv4.conf.enp1s0f0.rp_filter = 2 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout net.ipv4.conf.enp1s0f1.rp_filter = 2 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout net.ipv4.conf.enp7s0f4u2u2c2.rp_filter = 2 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout net.ipv4.conf.lo.rp_filter = 2 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout net.ipv4.conf.default.accept_source_route = 0 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout net.ipv4.conf.enp1s0f0.accept_source_route = 0 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout net.ipv4.conf.enp1s0f1.accept_source_route = 0 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout net.ipv4.conf.enp7s0f4u2u2c2.accept_source_route = 0 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout net.ipv4.conf.lo.accept_source_route = 0 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout net.ipv4.conf.default.promote_secondaries = 1 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout net.ipv4.conf.enp1s0f0.promote_secondaries = 1 2026-02-14 01:39:24,398 7faf01405e00 DEBUG sysctl: stdout net.ipv4.conf.enp1s0f1.promote_secondaries = 1 2026-02-14 01:39:24,399 7faf01405e00 DEBUG sysctl: stdout net.ipv4.conf.enp7s0f4u2u2c2.promote_secondaries = 1 2026-02-14 01:39:24,399 7faf01405e00 DEBUG sysctl: stdout net.ipv4.conf.lo.promote_secondaries = 1 2026-02-14 01:39:24,399 7faf01405e00 DEBUG sysctl: stdout net.ipv4.ping_group_range = 0 2147483647 2026-02-14 01:39:24,399 7faf01405e00 DEBUG sysctl: stdout net.core.default_qdisc = fq_codel 2026-02-14 01:39:24,399 7faf01405e00 DEBUG sysctl: stdout fs.protected_hardlinks = 1 2026-02-14 01:39:24,399 7faf01405e00 DEBUG sysctl: stdout fs.protected_symlinks = 1 2026-02-14 01:39:24,399 7faf01405e00 DEBUG sysctl: stdout fs.protected_regular = 1 2026-02-14 01:39:24,399 7faf01405e00 DEBUG sysctl: stdout fs.protected_fifos = 1 2026-02-14 01:39:24,399 7faf01405e00 DEBUG sysctl: stdout net.core.optmem_max = 81920 2026-02-14 01:39:24,399 7faf01405e00 DEBUG sysctl: stdout kernel.pid_max = 4194304 2026-02-14 01:39:24,399 7faf01405e00 DEBUG sysctl: stdout kernel.kptr_restrict = 1 2026-02-14 01:39:24,399 7faf01405e00 DEBUG sysctl: stdout net.ipv4.conf.default.rp_filter = 1 2026-02-14 01:39:24,399 7faf01405e00 DEBUG sysctl: stdout net.ipv4.conf.enp1s0f0.rp_filter = 1 2026-02-14 01:39:24,399 7faf01405e00 DEBUG sysctl: stdout net.ipv4.conf.enp1s0f1.rp_filter = 1 2026-02-14 01:39:24,399 7faf01405e00 DEBUG sysctl: stdout net.ipv4.conf.enp7s0f4u2u2c2.rp_filter = 1 2026-02-14 01:39:24,399 7faf01405e00 DEBUG sysctl: stdout net.ipv4.conf.lo.rp_filter = 1 2026-02-14 01:39:24,399 7faf01405e00 DEBUG sysctl: stdout fs.aio-max-nr = 2097152 2026-02-14 01:39:24,399 7faf01405e00 DEBUG sysctl: stdout fs.aio-max-nr = 1048576 2026-02-14 01:39:24,399 7faf01405e00 DEBUG sysctl: stdout kernel.pid_max = 4194304 2026-02-14 01:39:24,399 7faf01405e00 DEBUG sysctl: stdout kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-14 01:39:24,399 7faf01405e00 DEBUG sysctl: stdout kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-14 01:39:27,313 7faf01405e00 DEBUG firewalld.service is not enabled 2026-02-14 01:39:27,313 7faf01405e00 DEBUG Not possible to enable service . firewalld.service is not available 2026-02-14 01:39:27,451 7f1a0d1d0e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 01:39:28,566 7fc9b927fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:39:28,790 7fa2bebe9e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'list-networks'] 2026-02-14 01:39:28,944 7f4f2d26ee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:20ed745b2443ef2bcde9786bc2b320e2406866236ee87c83b3f1daa2d7eecdd4', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 01:39:28,960 7f4f2d26ee00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:39:28,960 7f4f2d26ee00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:40:31,405 7ffac804fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:41:06,392 7fc8f49d6e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'inspect-image'] 2026-02-14 01:41:06,428 7fc8f49d6e00 DEBUG /usr/bin/podman: stdout edc95c9460f10a90f1e921f9347538ed9be0b1c7aee54201d89be4c7dacb23c7,[quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9] 2026-02-14 01:41:06,554 7fc8f49d6e00 DEBUG ceph: stdout ceph version 20.2.0-667-geae49298 (eae49298735c216a37f3957204b0fe058b58e376) tentacle (stable) 2026-02-14 01:41:06,686 7f38bfa19e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:41:06,702 7f38bfa19e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'mds.cephfs.trial014.ubresr', 'image': '', 'deploy_arguments': [], 'params': {'allow_ptrace': True}, 'meta': {'service_name': 'mds.cephfs', 'ports': [], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[mds.cephfs.trial014.ubresr]\nkey = AQCB0Y9paSGPGBAAe4qWrzsf3JPgerwk3WRkCA==\n'}} 2026-02-14 01:41:06,702 7f38bfa19e00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9' 2026-02-14 01:41:06,716 7f38bfa19e00 INFO Redeploy daemon mds.cephfs.trial014.ubresr ... 2026-02-14 01:41:07,376 7f38bfa19e00 DEBUG firewalld.service is not enabled 2026-02-14 01:41:07,376 7f38bfa19e00 DEBUG Not possible to enable service . firewalld.service is not available 2026-02-14 01:41:07,511 7f25de241e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 01:41:08,587 7f2dd5a66e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:41:08,800 7f8beba2ee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'list-networks'] 2026-02-14 01:41:08,958 7fd6fb012e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 01:41:08,974 7fd6fb012e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:41:08,974 7fd6fb012e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:41:09,714 7f4c02e56e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'inspect-image'] 2026-02-14 01:41:09,749 7f4c02e56e00 DEBUG /usr/bin/podman: stdout edc95c9460f10a90f1e921f9347538ed9be0b1c7aee54201d89be4c7dacb23c7,[quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9] 2026-02-14 01:41:09,864 7f4c02e56e00 DEBUG ceph: stdout ceph version 20.2.0-667-geae49298 (eae49298735c216a37f3957204b0fe058b58e376) tentacle (stable) 2026-02-14 01:41:09,992 7f1f2cbf4e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:41:10,008 7f1f2cbf4e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'mds.cephfs.trial014.kwvbap', 'image': '', 'deploy_arguments': [], 'params': {'allow_ptrace': True}, 'meta': {'service_name': 'mds.cephfs', 'ports': [], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[mds.cephfs.trial014.kwvbap]\nkey = AQCC0Y9pGiBGMxAA7p0e98mGukEB03+oiEk3QQ==\n'}} 2026-02-14 01:41:10,008 7f1f2cbf4e00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9' 2026-02-14 01:41:10,021 7f1f2cbf4e00 INFO Redeploy daemon mds.cephfs.trial014.kwvbap ... 2026-02-14 01:41:10,697 7f1f2cbf4e00 DEBUG firewalld.service is not enabled 2026-02-14 01:41:10,697 7f1f2cbf4e00 DEBUG Not possible to enable service . firewalld.service is not available 2026-02-14 01:41:10,832 7f38e5091e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 01:41:11,924 7f5304c81e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:41:12,138 7f6edf04ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'list-networks'] 2026-02-14 01:41:12,292 7fdf37413e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 01:41:12,309 7fdf37413e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:41:12,309 7fdf37413e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:41:26,305 7fdcf27cae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'inspect-image'] 2026-02-14 01:41:26,340 7fdcf27cae00 DEBUG /usr/bin/podman: stdout edc95c9460f10a90f1e921f9347538ed9be0b1c7aee54201d89be4c7dacb23c7,[quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9] 2026-02-14 01:41:26,453 7fdcf27cae00 DEBUG ceph: stdout ceph version 20.2.0-667-geae49298 (eae49298735c216a37f3957204b0fe058b58e376) tentacle (stable) 2026-02-14 01:41:26,584 7f157cc60e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:41:26,601 7f157cc60e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'ceph-exporter.trial014', 'image': '', 'deploy_arguments': [], 'params': {'tcp_ports': [9926], 'allow_ptrace': True}, 'meta': {'service_name': 'ceph-exporter', 'ports': [9926], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'config': '# minimal ceph.conf for 29ce0a10-0945-11f1-b2be-d404e6e7d460\n[global]\n\tfsid = 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\tmon_host = [v2:10.20.193.14:3300/0,v1:10.20.193.14:6789/0] [v2:10.20.193.30:3300/0,v1:10.20.193.30:6789/0]\n', 'keyring': '[client.ceph-exporter.trial014]\nkey = AQAL0Y9pSQnMEBAAd1kOyDHxNKSk3p930enz8A==\n', 'prio-limit': '5', 'stats-period': '5'}} 2026-02-14 01:41:26,601 7f157cc60e00 DEBUG Determined image: 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9' 2026-02-14 01:41:26,615 7f157cc60e00 INFO Redeploy daemon ceph-exporter.trial014 ... 2026-02-14 01:41:26,746 7f157cc60e00 INFO Writing ceph-exporter config... 2026-02-14 01:41:27,336 7f157cc60e00 DEBUG firewalld.service is not enabled 2026-02-14 01:41:27,337 7f157cc60e00 DEBUG Not possible to open ports <[9926]>. firewalld.service is not available 2026-02-14 01:41:28,729 7fa88de76e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 01:41:29,722 7f6ab9631e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:41:29,937 7fd9a84a7e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'list-networks'] 2026-02-14 01:41:30,093 7f342b436e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 01:41:30,109 7f342b436e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:41:30,109 7f342b436e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:41:30,871 7fa8994d6e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'inspect-image'] 2026-02-14 01:41:30,905 7fa8994d6e00 DEBUG /usr/bin/podman: stdout edc95c9460f10a90f1e921f9347538ed9be0b1c7aee54201d89be4c7dacb23c7,[quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9] 2026-02-14 01:41:31,024 7fa8994d6e00 DEBUG ceph: stdout ceph version 20.2.0-667-geae49298 (eae49298735c216a37f3957204b0fe058b58e376) tentacle (stable) 2026-02-14 01:41:31,148 7fa7e9cd2e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.io/prometheus/node-exporter:v1.9.1', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:41:31,164 7fa7e9cd2e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'node-exporter.trial014', 'image': '', 'deploy_arguments': [], 'params': {'tcp_ports': [9100], 'allow_ptrace': True}, 'meta': {'service_name': 'node-exporter', 'ports': [9100], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {}} 2026-02-14 01:41:31,164 7fa7e9cd2e00 DEBUG Determined image: 'quay.io/prometheus/node-exporter:v1.9.1' 2026-02-14 01:41:31,177 7fa7e9cd2e00 INFO Redeploy daemon node-exporter.trial014 ... 2026-02-14 01:41:32,360 7fa7e9cd2e00 DEBUG firewalld.service is not enabled 2026-02-14 01:41:32,360 7fa7e9cd2e00 DEBUG Not possible to open ports <[9100]>. firewalld.service is not available 2026-02-14 01:41:34,282 7f389e7fbe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 01:41:35,292 7f786245ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:41:35,507 7f20dcc43e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'list-networks'] 2026-02-14 01:41:35,672 7f787448ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 01:41:35,688 7f787448ae00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:41:35,688 7f787448ae00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:41:36,433 7f23f27e1e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'inspect-image'] 2026-02-14 01:41:36,467 7f23f27e1e00 DEBUG /usr/bin/podman: stdout edc95c9460f10a90f1e921f9347538ed9be0b1c7aee54201d89be4c7dacb23c7,[quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9] 2026-02-14 01:41:36,586 7f23f27e1e00 DEBUG ceph: stdout ceph version 20.2.0-667-geae49298 (eae49298735c216a37f3957204b0fe058b58e376) tentacle (stable) 2026-02-14 01:41:36,859 7f296ddfae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.io/prometheus/prometheus:v3.6.0', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:41:36,875 7f296ddfae00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'prometheus.trial014', 'image': '', 'deploy_arguments': [], 'params': {'tcp_ports': [9095], 'allow_ptrace': True}, 'meta': {'service_name': 'prometheus', 'ports': [9095], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'files': {'prometheus.yml': "# This file is generated by cephadm.\nglobal:\n scrape_interval: 10s\n evaluation_interval: 10s\n external_labels:\n cluster: 29ce0a10-0945-11f1-b2be-d404e6e7d460\n\nrule_files:\n - /etc/prometheus/alerting/*\n\nalerting:\n alertmanagers:\n - scheme: http\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=alertmanager\n\nscrape_configs:\n - job_name: 'ceph'\n relabel_configs:\n - source_labels: [__address__]\n target_label: cluster\n replacement: 29ce0a10-0945-11f1-b2be-d404e6e7d460\n - source_labels: [instance]\n target_label: instance\n replacement: 'ceph_cluster'\n honor_labels: true\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=mgr-prometheus\n\n - job_name: 'node'\n relabel_configs:\n - source_labels: [__address__]\n target_label: cluster\n replacement: 29ce0a10-0945-11f1-b2be-d404e6e7d460\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=node-exporter\n\n\n - job_name: 'ceph-exporter'\n relabel_configs:\n - source_labels: [__address__]\n target_label: cluster\n replacement: 29ce0a10-0945-11f1-b2be-d404e6e7d460\n honor_labels: true\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=ceph-exporter\n\n - job_name: 'nvmeof'\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=nvmeof\n\n - job_name: 'nfs'\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=nfs\n\n - job_name: 'smb'\n http_sd_configs:\n - url: http://10.20.193.14:8765/sd/prometheus/sd-config?service=smb\n\n", '/etc/prometheus/alerting/ceph_alerts.yml': 'groups:\n - name: "cluster health"\n rules:\n - alert: "CephHealthError"\n annotations:\n description: "The cluster state has been HEALTH_ERROR for more than 5 minutes on cluster {{ $labels.cluster }}. Please check \'ceph health detail\' for more information."\n summary: "Ceph is in the ERROR state on cluster {{ $labels.cluster }}"\n expr: "ceph_health_status == 2"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.2.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephHealthWarning"\n annotations:\n description: "The cluster state has been HEALTH_WARN for more than 15 minutes on cluster {{ $labels.cluster }}. Please check \'ceph health detail\' for more information."\n summary: "Ceph is in the WARNING state on cluster {{ $labels.cluster }}"\n expr: "ceph_health_status == 1"\n for: "15m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "mon"\n rules:\n - alert: "CephMonDownQuorumAtRisk"\n annotations:\n description: "{{ $min := printf \\"floor(count(ceph_mon_metadata{cluster=\'%s\'}) / 2) + 1\\" .Labels.cluster | query | first | value }}Quorum requires a majority of monitors (x {{ $min }}) to be active. Without quorum the cluster will become inoperable, affecting all services and connected clients. The following monitors are down: {{- range printf \\"(ceph_mon_quorum_status{cluster=\'%s\'} == 0) + on(cluster,ceph_daemon) group_left(hostname) (ceph_mon_metadata * 0)\\" .Labels.cluster | query }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-down"\n summary: "Monitor quorum is at risk on cluster {{ $labels.cluster }}"\n expr: |\n (\n (ceph_health_detail{name="MON_DOWN"} == 1) * on() group_right(cluster) (\n count(ceph_mon_quorum_status == 1) by(cluster)== bool (floor(count(ceph_mon_metadata) by(cluster) / 2) + 1)\n )\n ) == 1\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.3.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephMonDown"\n annotations:\n description: "{{ $down := printf \\"count(ceph_mon_quorum_status{cluster=\'%s\'} == 0)\\" .Labels.cluster | query | first | value }}{{ $s := \\"\\" }}{{ if gt $down 1.0 }}{{ $s = \\"s\\" }}{{ end }}You have {{ $down }} monitor{{ $s }} down. Quorum is still intact, but the loss of an additional monitor will make your cluster inoperable. The following monitors are down: {{- range printf \\"(ceph_mon_quorum_status{cluster=\'%s\'} == 0) + on(cluster,ceph_daemon) group_left(hostname) (ceph_mon_metadata * 0)\\" .Labels.cluster | query }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-down"\n summary: "One or more monitors down on cluster {{ $labels.cluster }}"\n expr: |\n (count by (cluster) (ceph_mon_quorum_status == 0)) <= (count by (cluster) (ceph_mon_metadata) - floor((count by (cluster) (ceph_mon_metadata) / 2 + 1)))\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephMonDiskspaceCritical"\n annotations:\n description: "The free space available to a monitor\'s store is critically low. You should increase the space available to the monitor(s). The default directory is /var/lib/ceph/mon-*/data/store.db on traditional deployments, and /var/lib/rook/mon-*/data/store.db on the mon pod\'s worker node for Rook. Look for old, rotated versions of *.log and MANIFEST*. Do NOT touch any *.sst files. Also check any other directories under /var/lib/rook and other directories on the same filesystem, often /var/log and /var/tmp are culprits. Your monitor hosts are; {{- range query \\"ceph_mon_metadata\\"}} - {{ .Labels.hostname }} {{- end }}"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-disk-crit"\n summary: "Filesystem space on at least one monitor is critically low on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"MON_DISK_CRIT\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.3.2"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephMonDiskspaceLow"\n annotations:\n description: "The space available to a monitor\'s store is approaching full (>70% is the default). You should increase the space available to the monitor(s). The default directory is /var/lib/ceph/mon-*/data/store.db on traditional deployments, and /var/lib/rook/mon-*/data/store.db on the mon pod\'s worker node for Rook. Look for old, rotated versions of *.log and MANIFEST*. Do NOT touch any *.sst files. Also check any other directories under /var/lib/rook and other directories on the same filesystem, often /var/log and /var/tmp are culprits. Your monitor hosts are; {{- range query \\"ceph_mon_metadata\\"}} - {{ .Labels.hostname }} {{- end }}"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-disk-low"\n summary: "Drive space on at least one monitor is approaching full on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"MON_DISK_LOW\\"} == 1"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephMonClockSkew"\n annotations:\n description: "Ceph monitors rely on closely synchronized time to maintain quorum and cluster consistency. This event indicates that the time on at least one mon has drifted too far from the lead mon. Review cluster status with ceph -s. This will show which monitors are affected. Check the time sync status on each monitor host with \'ceph time-sync-status\' and the state and peers of your ntpd or chrony daemon."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-clock-skew"\n summary: "Clock skew detected among monitors on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"MON_CLOCK_SKEW\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "osd"\n rules:\n - alert: "CephOSDDownHigh"\n annotations:\n description: "{{ $value | humanize }}% or {{ with printf \\"count (ceph_osd_up{cluster=\'%s\'} == 0)\\" .Labels.cluster | query }}{{ . | first | value }}{{ end }} of {{ with printf \\"count (ceph_osd_up{cluster=\'%s\'})\\" .Labels.cluster | query }}{{ . | first | value }}{{ end }} OSDs are down (>= 10%). The following OSDs are down: {{- range printf \\"(ceph_osd_up{cluster=\'%s\'} * on(cluster, ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0\\" .Labels.cluster | query }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}"\n summary: "More than 10% of OSDs are down on cluster {{ $labels.cluster }}"\n expr: "count by (cluster) (ceph_osd_up == 0) / count by (cluster) (ceph_osd_up) * 100 >= 10"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephOSDHostDown"\n annotations:\n description: "The following OSDs are down: {{- range printf \\"(ceph_osd_up{cluster=\'%s\'} * on(cluster,ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0\\" .Labels.cluster | query }} - {{ .Labels.hostname }} : {{ .Labels.ceph_daemon }} {{- end }}"\n summary: "An OSD host is offline on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"OSD_HOST_DOWN\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.8"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDDown"\n annotations:\n description: "{{ $num := printf \\"count(ceph_osd_up{cluster=\'%s\'} == 0) \\" .Labels.cluster | query | first | value }}{{ $s := \\"\\" }}{{ if gt $num 1.0 }}{{ $s = \\"s\\" }}{{ end }}{{ $num }} OSD{{ $s }} down for over 5mins. The following OSD{{ $s }} {{ if eq $s \\"\\" }}is{{ else }}are{{ end }} down: {{- range printf \\"(ceph_osd_up{cluster=\'%s\'} * on(cluster,ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0\\" .Labels.cluster | query }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-down"\n summary: "An OSD has been marked down on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"OSD_DOWN\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.2"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDNearFull"\n annotations:\n description: "One or more OSDs have reached the NEARFULL threshold. Use \'ceph health detail\' and \'ceph osd df\' to identify the problem. To resolve, add capacity to the affected OSD\'s failure domain, restore down/out OSDs, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-nearfull"\n summary: "OSD(s) running low on free space (NEARFULL) on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"OSD_NEARFULL\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.3"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDFull"\n annotations:\n description: "An OSD has reached the FULL threshold. Writes to pools that share the affected OSD will be blocked. Use \'ceph health detail\' and \'ceph osd df\' to identify the problem. To resolve, add capacity to the affected OSD\'s failure domain, restore down/out OSDs, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-full"\n summary: "OSD full, writes blocked on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"OSD_FULL\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.6"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephOSDBackfillFull"\n annotations:\n description: "An OSD has reached the BACKFILL FULL threshold. This will prevent rebalance operations from completing. Use \'ceph health detail\' and \'ceph osd df\' to identify the problem. To resolve, add capacity to the affected OSD\'s failure domain, restore down/out OSDs, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-backfillfull"\n summary: "OSD(s) too full for backfill operations on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"OSD_BACKFILLFULL\\"} > 0"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDTooManyRepairs"\n annotations:\n description: "Reads from an OSD have used a secondary PG to return data to the client, indicating a potential failing drive."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-too-many-repairs"\n summary: "OSD reports a high number of read errors on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"OSD_TOO_MANY_REPAIRS\\"} == 1"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDTimeoutsPublicNetwork"\n annotations:\n description: "OSD heartbeats on the cluster\'s \'public\' network (frontend) are running slow. Investigate the network for latency or loss issues. Use \'ceph health detail\' to show the affected OSDs."\n summary: "Network issues delaying OSD heartbeats (public network) on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"OSD_SLOW_PING_TIME_FRONT\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDTimeoutsClusterNetwork"\n annotations:\n description: "OSD heartbeats on the cluster\'s \'cluster\' network (backend) are slow. Investigate the network for latency issues on this subnet. Use \'ceph health detail\' to show the affected OSDs."\n summary: "Network issues delaying OSD heartbeats (cluster network) on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"OSD_SLOW_PING_TIME_BACK\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDInternalDiskSizeMismatch"\n annotations:\n description: "One or more OSDs have an internal inconsistency between metadata and the size of the device. This could lead to the OSD(s) crashing in future. You should redeploy the affected OSDs."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#bluestore-disk-size-mismatch"\n summary: "OSD size inconsistency error on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"BLUESTORE_DISK_SIZE_MISMATCH\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephDeviceFailurePredicted"\n annotations:\n description: "The device health module has determined that one or more devices will fail soon. To review device status use \'ceph device ls\'. To show a specific device use \'ceph device info \'. Mark the OSD out so that data may migrate to other OSDs. Once the OSD has drained, destroy the OSD, replace the device, and redeploy the OSD."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#id2"\n summary: "Device(s) predicted to fail soon on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"DEVICE_HEALTH\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephDeviceFailurePredictionTooHigh"\n annotations:\n description: "The device health module has determined that devices predicted to fail can not be remediated automatically, since too many OSDs would be removed from the cluster to ensure performance and availability. Prevent data integrity issues by adding new OSDs so that data may be relocated."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#device-health-toomany"\n summary: "Too many devices are predicted to fail on cluster {{ $labels.cluster }}, unable to resolve"\n expr: "ceph_health_detail{name=\\"DEVICE_HEALTH_TOOMANY\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.7"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephDeviceFailureRelocationIncomplete"\n annotations:\n description: "The device health module has determined that one or more devices will fail soon, but the normal process of relocating the data on the device to other OSDs in the cluster is blocked. \\nEnsure that the cluster has available free space. It may be necessary to add capacity to the cluster to allow data from the failing device to successfully migrate, or to enable the balancer."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#device-health-in-use"\n summary: "Device failure is predicted, but unable to relocate data on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"DEVICE_HEALTH_IN_USE\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDFlapping"\n annotations:\n description: "OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked down and back up {{ $value | humanize }} times once a minute for 5 minutes. This may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster network, or the public network if no cluster network is deployed. Check the network stats on the listed host(s)."\n documentation: "https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds"\n summary: "Network issues are causing OSDs to flap (mark each other down) on cluster {{ $labels.cluster }}"\n expr: "(rate(ceph_osd_up[5m]) * on(cluster,ceph_daemon) group_left(hostname) ceph_osd_metadata) * 60 > 1"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.4"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephOSDReadErrors"\n annotations:\n description: "An OSD has encountered read errors, but the OSD has recovered by retrying the reads. This may indicate an issue with hardware or the kernel."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#bluestore-spurious-read-errors"\n summary: "Device read errors detected on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"BLUESTORE_SPURIOUS_READ_ERRORS\\"} == 1"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPGImbalance"\n annotations:\n description: "OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} deviates by more than 30% from average PG count in the device class {{ $labels.device_class }}."\n summary: "PGs are not balanced across OSDs on cluster {{ $labels.cluster }}"\n expr: |\n abs(\n (\n (\n (ceph_osd_numpg > 0)\n * on (cluster, job, ceph_daemon) group_left(hostname, device_class) ceph_osd_metadata\n )\n - on (cluster, job, device_class) group_left avg(\n (ceph_osd_numpg > 0)\n * on (cluster, job, ceph_daemon) group_left(hostname, device_class) ceph_osd_metadata\n ) by (cluster, job, device_class)\n )\n / on (cluster, job, device_class) group_left avg(\n (ceph_osd_numpg > 0)\n * on (cluster, job, ceph_daemon) group_left(hostname, device_class) ceph_osd_metadata\n ) by (cluster, job, device_class)\n ) > 0.30\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.4.5"\n severity: "warning"\n type: "ceph_default"\n - name: "mds"\n rules:\n - alert: "CephFilesystemDamaged"\n annotations:\n description: "Filesystem metadata has been corrupted. Data may be inaccessible. Analyze metrics from the MDS daemon admin socket, or escalate to support."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages#cephfs-health-messages"\n summary: "CephFS filesystem is damaged on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"MDS_DAMAGE\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephFilesystemOffline"\n annotations:\n description: "All MDS ranks are unavailable. The MDS daemons managing metadata are down, rendering the filesystem offline."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-all-down"\n summary: "CephFS filesystem is offline on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"MDS_ALL_DOWN\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.3"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephFilesystemDegraded"\n annotations:\n description: "One or more metadata daemons (MDS ranks) are failed or in a damaged state. At best the filesystem is partially available, at worst the filesystem is completely unusable."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#fs-degraded"\n summary: "CephFS filesystem is degraded on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"FS_DEGRADED\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.4"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephFilesystemMDSRanksLow"\n annotations:\n description: "The filesystem\'s \'max_mds\' setting defines the number of MDS ranks in the filesystem. The current number of active MDS daemons is less than this value."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-up-less-than-max"\n summary: "Ceph MDS daemon count is lower than configured on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"MDS_UP_LESS_THAN_MAX\\"} > 0"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephFilesystemInsufficientStandby"\n annotations:\n description: "The minimum number of standby daemons required by standby_count_wanted is less than the current number of standby daemons. Adjust the standby count or increase the number of MDS daemons."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-insufficient-standby"\n summary: "Ceph filesystem standby daemons too few on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"MDS_INSUFFICIENT_STANDBY\\"} > 0"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephFilesystemFailureNoStandby"\n annotations:\n description: "An MDS daemon has failed, leaving only one active rank and no available standby. Investigate the cause of the failure or add a standby MDS."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#fs-with-failed-mds"\n summary: "MDS daemon failed, no further standby available on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"FS_WITH_FAILED_MDS\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.5"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephFilesystemReadOnly"\n annotations:\n description: "The filesystem has switched to READ ONLY due to an unexpected error when writing to the metadata pool. Either analyze the output from the MDS daemon admin socket, or escalate to support."\n documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages#cephfs-health-messages"\n summary: "CephFS filesystem in read only mode due to write error(s) on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"MDS_HEALTH_READ_ONLY\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.5.2"\n severity: "critical"\n type: "ceph_default"\n - name: "mgr"\n rules:\n - alert: "CephMgrModuleCrash"\n annotations:\n description: "One or more mgr modules have crashed and have yet to be acknowledged by an administrator. A crashed module may impact functionality within the cluster. Use the \'ceph crash\' command to determine which module has failed, and archive it to acknowledge the failure."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#recent-mgr-module-crash"\n summary: "A manager module has recently crashed on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"RECENT_MGR_MODULE_CRASH\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.6.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephMgrPrometheusModuleInactive"\n annotations:\n description: "The mgr/prometheus module at {{ $labels.instance }} is unreachable. This could mean that the module has been disabled or the mgr daemon itself is down. Without the mgr/prometheus module metrics and alerts will no longer function. Open a shell to an admin node or toolbox pod and use \'ceph -s\' to to determine whether the mgr is active. If the mgr is not active, restart it, otherwise you can determine module status with \'ceph mgr module ls\'. If it is not listed as enabled, enable it with \'ceph mgr module enable prometheus\'."\n summary: "The mgr/prometheus module is not available"\n expr: "up{job=\\"ceph\\"} == 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.6.2"\n severity: "critical"\n type: "ceph_default"\n - name: "pgs"\n rules:\n - alert: "CephPGsInactive"\n annotations:\n description: "{{ $value }} PGs have been inactive for more than 5 minutes in pool {{ $labels.name }}. Inactive placement groups are not able to serve read/write requests."\n summary: "One or more placement groups are inactive on cluster {{ $labels.cluster }}"\n expr: "ceph_pool_metadata * on(cluster,pool_id,instance) group_left() (ceph_pg_total - ceph_pg_active) > 0"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGsUnclean"\n annotations:\n description: "{{ $value }} PGs have been unclean for more than 15 minutes in pool {{ $labels.name }}. Unclean PGs have not recovered from a previous failure."\n summary: "One or more placement groups are marked unclean on cluster {{ $labels.cluster }}"\n expr: "ceph_pool_metadata * on(cluster,pool_id,instance) group_left() (ceph_pg_total - ceph_pg_clean) > 0"\n for: "15m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.2"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPGsDamaged"\n annotations:\n description: "During data consistency checks (scrub), at least one PG has been flagged as being damaged or inconsistent. Check to see which PG is affected, and attempt a manual repair if necessary. To list problematic placement groups, use \'rados list-inconsistent-pg \'. To repair PGs use the \'ceph pg repair \' command."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-damaged"\n summary: "Placement group damaged, manual intervention needed on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=~\\"PG_DAMAGED|OSD_SCRUB_ERRORS\\"} == 1"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.4"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGRecoveryAtRisk"\n annotations:\n description: "Data redundancy is at risk since one or more OSDs are at or above the \'full\' threshold. Add more capacity to the cluster, restore down/out OSDs, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-recovery-full"\n summary: "OSDs are too full for recovery on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"PG_RECOVERY_FULL\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.5"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGUnavailableBlockingIO"\n annotations:\n description: "Data availability is reduced, impacting the cluster\'s ability to service I/O. One or more placement groups (PGs) are in a state that blocks I/O."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-availability"\n summary: "PG is unavailable on cluster {{ $labels.cluster }}, blocking I/O"\n expr: "((ceph_health_detail{name=\\"PG_AVAILABILITY\\"} == 1) - scalar(ceph_health_detail{name=\\"OSD_DOWN\\"})) == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.3"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGBackfillAtRisk"\n annotations:\n description: "Data redundancy may be at risk due to lack of free space within the cluster. One or more OSDs have reached the \'backfillfull\' threshold. Add more capacity, or delete unwanted data."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-backfill-full"\n summary: "Backfill operations are blocked due to lack of free space on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"PG_BACKFILL_FULL\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.7.6"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPGNotScrubbed"\n annotations:\n description: "One or more PGs have not been scrubbed recently. Scrubs check metadata integrity, protecting against bit-rot. They check that metadata is consistent across data replicas. When PGs miss their scrub interval, it may indicate that the scrub window is too small, or PGs were not in a \'clean\' state during the scrub window. You can manually initiate a scrub with: ceph pg scrub "\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-not-scrubbed"\n summary: "Placement group(s) have not been scrubbed on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"PG_NOT_SCRUBBED\\"} == 1"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPGsHighPerOSD"\n annotations:\n description: "The number of placement groups per OSD is too high (exceeds the mon_max_pg_per_osd setting).\\n Check that the pg_autoscaler has not been disabled for any pools with \'ceph osd pool autoscale-status\', and that the profile selected is appropriate. You may also adjust the target_size_ratio of a pool to guide the autoscaler based on the expected relative size of the pool (\'ceph osd pool set cephfs.cephfs.meta target_size_ratio .1\') or set the pg_autoscaler mode to \'warn\' and adjust pg_num appropriately for one or more pools."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks/#too-many-pgs"\n summary: "Placement groups per OSD is too high on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"TOO_MANY_PGS\\"} == 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPGNotDeepScrubbed"\n annotations:\n description: "One or more PGs have not been deep scrubbed recently. Deep scrubs protect against bit-rot. They compare data replicas to ensure consistency. When PGs miss their deep scrub interval, it may indicate that the window is too small or PGs were not in a \'clean\' state during the deep-scrub window."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-not-deep-scrubbed"\n summary: "Placement group(s) have not been deep scrubbed on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"PG_NOT_DEEP_SCRUBBED\\"} == 1"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "nodes"\n rules:\n - alert: "CephNodeRootFilesystemFull"\n annotations:\n description: "Root volume is dangerously full: {{ $value | humanize }}% free."\n summary: "Root filesystem is dangerously full"\n expr: "node_filesystem_avail_bytes{mountpoint=\\"/\\"} / node_filesystem_size_bytes{mountpoint=\\"/\\"} * 100 < 5"\n for: "5m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.8.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephNodeNetworkPacketDrops"\n annotations:\n description: "Node {{ $labels.instance }} experiences packet drop > 0.5% or > 10 packets/s on interface {{ $labels.device }}."\n summary: "One or more NICs reports packet drops"\n expr: |\n (\n rate(node_network_receive_drop_total{device!="lo"}[1m]) +\n rate(node_network_transmit_drop_total{device!="lo"}[1m])\n ) / (\n rate(node_network_receive_packets_total{device!="lo"}[1m]) +\n rate(node_network_transmit_packets_total{device!="lo"}[1m])\n ) >= 0.0050000000000000001 and (\n rate(node_network_receive_drop_total{device!="lo"}[1m]) +\n rate(node_network_transmit_drop_total{device!="lo"}[1m])\n ) >= 10\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.8.2"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephNodeNetworkPacketErrors"\n annotations:\n description: "Node {{ $labels.instance }} experiences packet errors > 0.01% or > 10 packets/s on interface {{ $labels.device }}."\n summary: "One or more NICs reports packet errors on cluster {{ $labels.cluster }}"\n expr: |\n (\n rate(node_network_receive_errs_total{device!="lo"}[1m]) +\n rate(node_network_transmit_errs_total{device!="lo"}[1m])\n ) / (\n rate(node_network_receive_packets_total{device!="lo"}[1m]) +\n rate(node_network_transmit_packets_total{device!="lo"}[1m])\n ) >= 0.0001 or (\n rate(node_network_receive_errs_total{device!="lo"}[1m]) +\n rate(node_network_transmit_errs_total{device!="lo"}[1m])\n ) >= 10\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.8.3"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephNodeNetworkBondDegraded"\n annotations:\n description: "Bond {{ $labels.master }} is degraded on Node {{ $labels.instance }}."\n summary: "Degraded Bond on Node {{ $labels.instance }} on cluster {{ $labels.cluster }}"\n expr: |\n node_bonding_slaves - node_bonding_active != 0\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephNodeDiskspaceWarning"\n annotations:\n description: "Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will be full in less than 5 days based on the 48 hour trailing fill rate."\n summary: "Host filesystem free space is getting low on cluster {{ $labels.cluster }}"\n expr: "predict_linear(node_filesystem_free_bytes{device=~\\"/.*\\"}[2d], 3600 * 24 * 5) * on(cluster, instance) group_left(nodename) node_uname_info < 0"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.8.4"\n severity: "warning"\n type: "ceph_default"\n - alert: CephNodeInconsistentMTU\n expr: |\n node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)\n != on (cluster, device) group_left\n quantile by (cluster, device) (\n 0.5, node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)\n )\n labels:\n severity: warning\n type: ceph_default\n annotations:\n summary: "Node {{ $labels.instance }} has inconsistent MTU settings in cluster {{ $labels.cluster }}"\n description: "Network interface {{ $labels.device }} on node {{ $labels.instance }} has MTU {{ $value }} which differs from the cluster median."\n impact: |\n - May cause packet fragmentation or packet drops\n - Risk of degraded cluster communication and performance\n - Potential instability in services relying on consistent networking (e.g., Ceph, Kubernetes)\n fix: |\n - Check the MTU of interface `{{ $labels.device }}` on node `{{ $labels.instance }}`:\n ip link show {{ $labels.device }}\n\n - Find the median MTU value across the cluster by running this PromQL query in Prometheus:\n quantile by (cluster, device) (0.5, node_network_mtu_bytes * (node_network_up{device!="lo"} > 0))\n\n - Standardize MTU across all nodes to match the median (commonly 1500 or 9000):\n ip link set dev {{ $labels.device }} mtu \n\n - Make MTU setting persistent:\n - RHEL/CentOS: edit `/etc/sysconfig/network-scripts/ifcfg-`\n - Debian/Ubuntu: edit `/etc/netplan/*.yaml` and apply with `netplan apply`\n\n - Restart the affected interface or node if required.\n - name: "pools"\n rules:\n - alert: "CephPoolGrowthWarning"\n annotations:\n description: "Pool \'{{ $labels.name }}\' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours."\n summary: "Pool growth rate may soon exceed capacity on cluster {{ $labels.cluster }}"\n expr: "(predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(cluster,pool_id, instance) group_right() ceph_pool_metadata) >= 95"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.9.2"\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPoolBackfillFull"\n annotations:\n description: "A pool is approaching the near full threshold, which will prevent recovery/backfill operations from completing. Consider adding more capacity."\n summary: "Free space in a pool is too low for recovery/backfill on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"POOL_BACKFILLFULL\\"} > 0"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephPoolFull"\n annotations:\n description: "A pool has reached its MAX quota, or OSDs supporting the pool have reached the FULL threshold. Until this is resolved, writes to the pool will be blocked. Pool Breakdown (top 5) {{- range printf \\"topk(5, sort_desc(ceph_pool_percent_used{cluster=\'%s\'} * on(cluster,pool_id) group_right ceph_pool_metadata))\\" .Labels.cluster | query }} - {{ .Labels.name }} at {{ .Value }}% {{- end }} Increase the pool\'s quota, or add capacity to the cluster first then increase the pool\'s quota (e.g. ceph osd pool set quota max_bytes )"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pool-full"\n summary: "Pool is full - writes are blocked on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"POOL_FULL\\"} > 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.9.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephPoolNearFull"\n annotations:\n description: "A pool has exceeded the warning (percent full) threshold, or OSDs supporting the pool have reached the NEARFULL threshold. Writes may continue, but you are at risk of the pool going read-only if more capacity isn\'t made available. Determine the affected pool with \'ceph df detail\', looking at QUOTA BYTES and STORED. Increase the pool\'s quota, or add capacity to the cluster first then increase the pool\'s quota (e.g. ceph osd pool set quota max_bytes ). Also ensure that the balancer is active."\n summary: "One or more Ceph pools are nearly full on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"POOL_NEAR_FULL\\"} > 0"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "healthchecks"\n rules:\n - alert: "CephSlowOps"\n annotations:\n description: "{{ $value }} OSD requests are taking too long to process (osd_op_complaint_time exceeded)"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#slow-ops"\n summary: "OSD operations are slow to complete on cluster {{ $labels.cluster }}"\n expr: "ceph_healthcheck_slow_ops > 0"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "CephDaemonSlowOps"\n annotations:\n description: "{{ $labels.ceph_daemon }} operations are taking too long to process (complaint time exceeded)"\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#slow-ops"\n summary: "{{ $labels.ceph_daemon }} operations are slow to complete on cluster {{ $labels.cluster }}"\n expr: "ceph_daemon_health_metrics{type=\\"SLOW_OPS\\"} > 0"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "cephadm"\n rules:\n - alert: "CephadmUpgradeFailed"\n annotations:\n description: "The cephadm cluster upgrade process has failed. The cluster remains in an undetermined state. Please review the cephadm logs, to understand the nature of the issue"\n summary: "Ceph version upgrade has failed on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"UPGRADE_EXCEPTION\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.11.2"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephadmDaemonFailed"\n annotations:\n description: "A daemon managed by cephadm is no longer active. Determine, which daemon is down with \'ceph health detail\'. you may start daemons with the \'ceph orch daemon start \'"\n summary: "A ceph daemon managed by cephadm is down on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"CEPHADM_FAILED_DAEMON\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.11.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephadmPaused"\n annotations:\n description: "Cluster management has been paused manually. This will prevent the orchestrator from service management and reconciliation. If this is not intentional, resume cephadm operations with \'ceph orch resume\'"\n documentation: "https://docs.ceph.com/en/latest/cephadm/operations#cephadm-paused"\n summary: "Orchestration tasks via cephadm are PAUSED on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"CEPHADM_PAUSED\\"} > 0"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - name: "hardware"\n rules:\n - alert: "HardwareStorageError"\n annotations:\n description: "Some storage devices are in error. Check `ceph health detail`."\n summary: "Storage devices error(s) detected on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"HARDWARE_STORAGE\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.1"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwareMemoryError"\n annotations:\n description: "DIMM error(s) detected. Check `ceph health detail`."\n summary: "DIMM error(s) detected on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"HARDWARE_MEMORY\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.2"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwareProcessorError"\n annotations:\n description: "Processor error(s) detected. Check `ceph health detail`."\n summary: "Processor error(s) detected on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"HARDWARE_PROCESSOR\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.3"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwareNetworkError"\n annotations:\n description: "Network error(s) detected. Check `ceph health detail`."\n summary: "Network error(s) detected on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"HARDWARE_NETWORK\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.4"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwarePowerError"\n annotations:\n description: "Power supply error(s) detected. Check `ceph health detail`."\n summary: "Power supply error(s) detected on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"HARDWARE_POWER\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.5"\n severity: "critical"\n type: "ceph_default"\n - alert: "HardwareFanError"\n annotations:\n description: "Fan error(s) detected. Check `ceph health detail`."\n summary: "Fan error(s) detected on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"HARDWARE_FANS\\"} > 0"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.13.6"\n severity: "critical"\n type: "ceph_default"\n - name: "PrometheusServer"\n rules:\n - alert: "PrometheusJobMissing"\n annotations:\n description: "The prometheus job that scrapes from Ceph is no longer defined, this will effectively mean you\'ll have no metrics or alerts for the cluster. Please review the job definitions in the prometheus.yml file of the prometheus instance."\n summary: "The scrape job for Ceph is missing from Prometheus"\n expr: "absent(up{job=\\"ceph\\"})"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.12.1"\n severity: "critical"\n type: "ceph_default"\n - name: "rados"\n rules:\n - alert: "CephObjectMissing"\n annotations:\n description: "The latest version of a RADOS object can not be found, even though all OSDs are up. I/O requests for this object from clients will block (hang). Resolving this issue may require the object to be rolled back to a prior version manually, and manually verified."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#object-unfound"\n summary: "Object(s) marked UNFOUND on cluster {{ $labels.cluster }}"\n expr: "(ceph_health_detail{name=\\"OBJECT_UNFOUND\\"} == 1) * on() group_right(cluster) (count(ceph_osd_up == 1) by (cluster) == bool count(ceph_osd_metadata) by(cluster)) == 1"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.1"\n severity: "critical"\n type: "ceph_default"\n - name: "generic"\n rules:\n - alert: "CephDaemonCrash"\n annotations:\n description: "One or more daemons have crashed recently, and need to be acknowledged. This notification ensures that software crashes do not go unseen. To acknowledge a crash, use the \'ceph crash archive \' command."\n documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks/#recent-crash"\n summary: "One or more Ceph daemons have crashed, and are pending acknowledgement on cluster {{ $labels.cluster }}"\n expr: "ceph_health_detail{name=\\"RECENT_CRASH\\"} == 1"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.1.2"\n severity: "critical"\n type: "ceph_default"\n - name: "rbdmirror"\n rules:\n - alert: "CephRBDMirrorImagesPerDaemonHigh"\n annotations:\n description: "Number of image replications per daemon is not supposed to go beyond threshold 100"\n summary: "Number of image replications are now above 100 on cluster {{ $labels.cluster }}"\n expr: "sum by (cluster, ceph_daemon, namespace) (ceph_rbd_mirror_snapshot_image_snapshots) > 100"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.2"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephRBDMirrorImagesNotInSync"\n annotations:\n description: "Both local and remote RBD mirror images should be in sync."\n summary: "Some of the RBD mirror images are not in sync with the remote counter parts on cluster {{ $labels.cluster }}"\n expr: "sum by (cluster, ceph_daemon, image, namespace, pool) (topk by (cluster, ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_local_timestamp) - topk by (cluster, ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_remote_timestamp)) != 0"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.3"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephRBDMirrorImagesNotInSyncVeryHigh"\n annotations:\n description: "More than 10% of the images have synchronization problems."\n summary: "Number of unsynchronized images are very high on cluster {{ $labels.cluster }}"\n expr: "count by (ceph_daemon, cluster) ((topk by (cluster, ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_local_timestamp) - topk by (cluster, ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_remote_timestamp)) != 0) > (sum by (ceph_daemon, cluster) (ceph_rbd_mirror_snapshot_snapshots)*.1)"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.4"\n severity: "critical"\n type: "ceph_default"\n - alert: "CephRBDMirrorImageTransferBandwidthHigh"\n annotations:\n description: "Detected a heavy increase in bandwidth for rbd replications (over 80%) in the last 30 min. This might not be a problem, but it is good to review the number of images being replicated simultaneously"\n summary: "The replication network usage on cluster {{ $labels.cluster }} has been increased over 80% in the last 30 minutes. Review the number of images being replicated. This alert will be cleaned automatically after 30 minutes"\n expr: "rate(ceph_rbd_mirror_journal_replay_bytes[30m]) > 0.80"\n for: "1m"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.10.5"\n severity: "warning"\n type: "ceph_default"\n - name: "nvmeof"\n rules:\n - alert: "NVMeoFSubsystemNamespaceLimit"\n annotations:\n description: "Subsystems have a max namespace limit defined at creation time. This alert means that no more namespaces can be added to {{ $labels.nqn }}"\n summary: "{{ $labels.nqn }} subsystem has reached its maximum number of namespaces on cluster {{ $labels.cluster }}"\n expr: "(count by(nqn, cluster, instance) (ceph_nvmeof_subsystem_namespace_metadata)) >= on(nqn, instance) group_right(cluster) ceph_nvmeof_subsystem_namespace_limit"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFMultipleNamespacesOfRBDImage"\n annotations:\n description: "Each NVMeoF namespace must have a unique RBD pool and image, across all different gateway groups."\n summary: "RBD image {{ $labels.pool_name }}/{{ $labels.rbd_name }} cannot be reused for multiple NVMeoF namespace "\n expr: "count by(pool_name, rbd_name) (count by(bdev_name, pool_name, rbd_name) (ceph_nvmeof_bdev_metadata and on (bdev_name) ceph_nvmeof_subsystem_namespace_metadata)) > 1"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFTooManyGateways"\n annotations:\n description: "You may create many gateways, but 32 is the tested limit"\n summary: "Max supported gateways exceeded on cluster {{ $labels.cluster }}"\n expr: "count(ceph_nvmeof_gateway_info) by (cluster) > 32.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFMaxGatewayGroupSize"\n annotations:\n description: "You may create many gateways in a gateway group, but 8 is the tested limit"\n summary: "Max gateways within a gateway group ({{ $labels.group }}) exceeded on cluster {{ $labels.cluster }}"\n expr: "count(ceph_nvmeof_gateway_info) by (cluster,group) > 8.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFMaxGatewayGroups"\n annotations:\n description: "You may create many gateway groups, but 4 is the tested limit"\n summary: "Max gateway groups exceeded on cluster {{ $labels.cluster }}"\n expr: "count(count by (group, cluster) (ceph_nvmeof_gateway_info)) by (cluster) > 4.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFSingleGateway"\n annotations:\n description: "Although a single member gateway group is valid, it should only be used for test purposes"\n summary: "The gateway group {{ $labels.group }} consists of a single gateway - HA is not possible on cluster {{ $labels.cluster }}"\n expr: "count(ceph_nvmeof_gateway_info) by(cluster,group) == 1"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighGatewayCPU"\n annotations:\n description: "Typically, high CPU may indicate degraded performance. Consider increasing the number of reactor cores"\n summary: "CPU used by {{ $labels.instance }} NVMe-oF Gateway is high on cluster {{ $labels.cluster }}"\n expr: "label_replace(avg by(instance, cluster) (rate(ceph_nvmeof_reactor_seconds_total{mode=\\"busy\\"}[1m])),\\"instance\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\") > 80.00"\n for: "10m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFGatewayOpenSecurity"\n annotations:\n description: "It is good practice to ensure subsystems use host security to reduce the risk of unexpected data loss"\n summary: "Subsystem {{ $labels.nqn }} has been defined without host level security on cluster {{ $labels.cluster }}"\n expr: "ceph_nvmeof_subsystem_metadata{allow_any_host=\\"yes\\"}"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFTooManySubsystems"\n annotations:\n description: "NVMeoF gateway {{ $labels.gateway_host }} has reached or exceeded the supported maximum of 128 subsystems. Current count: {{ $value }}."\n summary: "The number of subsystems defined to the NVMeoF gateway reached or exceeded the supported values on cluster {{ $labels.cluster }}"\n expr: "count by(gateway_host, cluster) (label_replace(ceph_nvmeof_subsystem_metadata,\\"gateway_host\\",\\"$1\\",\\"instance\\",\\"(.*?)(?::.*)?\\")) >= 128.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFTooManyNamespaces"\n annotations:\n description: "NVMeoF gateway {{ $labels.gateway_host }} has reached or exceeded the supported maximum of 4096 namespaces. Current count: {{ $value }}."\n summary: "The number of namespaces defined to the NVMeoF gateway reached or exceeded supported values on cluster {{ $labels.cluster }}"\n expr: "sum by(gateway_host, cluster) (label_replace(ceph_nvmeof_subsystem_namespace_count,\\"gateway_host\\",\\"$1\\",\\"instance\\",\\"(.*?)(?::.*)?\\")) >= 4096.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFVersionMismatch"\n annotations:\n description: "This may indicate an issue with deployment. Check cephadm logs"\n summary: "Too many different NVMe-oF gateway releases active on cluster {{ $labels.cluster }}"\n expr: "count(count(ceph_nvmeof_gateway_info) by (cluster, version)) by (cluster) > 1"\n for: "1h"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighClientCount"\n annotations:\n description: "The supported limit for clients connecting to a subsystem is 128"\n summary: "The number of clients connected to {{ $labels.nqn }} is too high on cluster {{ $labels.cluster }}"\n expr: "ceph_nvmeof_subsystem_host_count > 128.00"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFMissingListener"\n annotations:\n description: "For every subsystem, each gateway should have a listener to balance traffic between gateways."\n summary: "No listener added for {{ $labels.instance }} NVMe-oF Gateway to {{ $labels.nqn }} subsystem"\n expr: "ceph_nvmeof_subsystem_listener_count == 0 and on(nqn) sum(ceph_nvmeof_subsystem_listener_count) by (nqn) > 0"\n for: "10m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFZeroListenerSubsystem"\n annotations:\n description: "NVMeoF gateway configuration incomplete; one of the subsystems have zero listeners."\n summary: "No listeners added to {{ $labels.nqn }} subsystem"\n expr: "sum(ceph_nvmeof_subsystem_listener_count) by (nqn) == 0"\n for: "10m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighHostCPU"\n annotations:\n description: "High CPU on a gateway host can lead to CPU contention and performance degradation"\n summary: "The CPU is high ({{ $value }}%) on NVMeoF Gateway host ({{ $labels.host }}) on cluster {{ $labels.cluster }}"\n expr: "100-((100*(avg by(cluster,host) (label_replace(rate(node_cpu_seconds_total{mode=\\"idle\\"}[5m]),\\"host\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\")) * on(cluster, host) group_right label_replace(ceph_nvmeof_gateway_info,\\"host\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\")))) >= 80.00"\n for: "10m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFInterfaceDown"\n annotations:\n description: "A NIC used by one or more subsystems is in a down state"\n summary: "Network interface {{ $labels.device }} is down on cluster {{ $labels.cluster }}"\n expr: "ceph_nvmeof_subsystem_listener_iface_info{operstate=\\"down\\"}"\n for: "30s"\n labels:\n oid: "1.3.6.1.4.1.50495.1.2.1.14.1"\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFInterfaceDuplex"\n annotations:\n description: "Until this is resolved, performance from the gateway will be degraded"\n summary: "Network interface {{ $labels.device }} is not running in full duplex mode on cluster {{ $labels.cluster }}"\n expr: "ceph_nvmeof_subsystem_listener_iface_info{duplex!=\\"full\\"}"\n for: "30s"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighReadLatency"\n annotations:\n description: "High latencies may indicate a constraint within the cluster e.g. CPU, network. Please investigate"\n summary: "The average read latency over the last 5 mins has reached 10 ms or more on {{ $labels.gateway }}"\n expr: "label_replace((avg by(instance) ((rate(ceph_nvmeof_bdev_read_seconds_total[1m]) / rate(ceph_nvmeof_bdev_reads_completed_total[1m])))),\\"gateway\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\") > 0.01"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHighWriteLatency"\n annotations:\n description: "High latencies may indicate a constraint within the cluster e.g. CPU, network. Please investigate"\n summary: "The average write latency over the last 5 mins has reached 20 ms or more on {{ $labels.gateway }}"\n expr: "label_replace((avg by(instance) ((rate(ceph_nvmeof_bdev_write_seconds_total[5m]) / rate(ceph_nvmeof_bdev_writes_completed_total[5m])))),\\"gateway\\",\\"$1\\",\\"instance\\",\\"(.*):.*\\") > 0.02"\n for: "5m"\n labels:\n severity: "warning"\n type: "ceph_default"\n - alert: "NVMeoFHostKeepAliveTimeout"\n annotations:\n description: "Host was disconnected due to host keep alive timeout"\n summary: "Host ({{ $labels.host_nqn }}) was disconnected {{ $value }} times from subsystem ({{ $labels.nqn }}) in last 24 hours"\n expr: "ceil(changes(ceph_nvmeof_host_keepalive_timeout[24h:]) / 2) > 0"\n for: "1m"\n labels:\n severity: "warning"\n type: "ceph_default"\n', '/etc/prometheus/alerting/custom_alerts.yml': ''}, 'retention_time': '15d', 'retention_size': '0', 'ip_to_bind_to': '', 'use_url_prefix': False}} 2026-02-14 01:41:36,875 7f296ddfae00 DEBUG Determined image: 'quay.io/prometheus/prometheus:v3.6.0' 2026-02-14 01:41:36,889 7f296ddfae00 INFO Redeploy daemon prometheus.trial014 ... 2026-02-14 01:41:39,998 7f296ddfae00 DEBUG firewalld.service is not enabled 2026-02-14 01:41:39,998 7f296ddfae00 DEBUG Not possible to open ports <[9095]>. firewalld.service is not available 2026-02-14 01:41:40,134 7ff8bb4c3e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 01:41:41,132 7f312106fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:41:41,350 7f914d06ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'list-networks'] 2026-02-14 01:41:41,507 7f67fb82ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 01:41:41,524 7f67fb82ce00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:41:41,524 7f67fb82ce00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:41:42,261 7fa25da6fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'inspect-image'] 2026-02-14 01:41:42,296 7fa25da6fe00 DEBUG /usr/bin/podman: stdout edc95c9460f10a90f1e921f9347538ed9be0b1c7aee54201d89be4c7dacb23c7,[quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9] 2026-02-14 01:41:42,414 7fa25da6fe00 DEBUG ceph: stdout ceph version 20.2.0-667-geae49298 (eae49298735c216a37f3957204b0fe058b58e376) tentacle (stable) 2026-02-14 01:41:42,541 7f807cc7ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.io/prometheus/alertmanager:v0.28.1', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:41:42,557 7f807cc7ae00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'alertmanager.trial014', 'image': '', 'deploy_arguments': [], 'params': {'tcp_ports': [9093, 9094], 'allow_ptrace': True}, 'meta': {'service_name': 'alertmanager', 'ports': [9093, 9094], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'files': {'alertmanager.yml': "# This file is generated by cephadm.\n# See https://prometheus.io/docs/alerting/configuration/ for documentation.\n\nglobal:\n resolve_timeout: 5m\n http_config:\n tls_config:\n insecure_skip_verify: true\n\nroute:\n receiver: 'default'\n routes:\n - group_by: ['alertname']\n group_wait: 10s\n group_interval: 10s\n repeat_interval: 1h\n receiver: 'ceph-dashboard'\n\nreceivers:\n- name: 'default'\n webhook_configs:\n- name: 'ceph-dashboard'\n webhook_configs:\n - url: 'https://trial014:8443/api/prometheus_receiver'\n - url: 'https://trial030:8443/api/prometheus_receiver'\n"}, 'peers': ['trial014:9094'], 'use_url_prefix': False, 'ip_to_bind_to': ''}} 2026-02-14 01:41:42,557 7f807cc7ae00 DEBUG Determined image: 'quay.io/prometheus/alertmanager:v0.28.1' 2026-02-14 01:41:42,571 7f807cc7ae00 INFO Redeploy daemon alertmanager.trial014 ... 2026-02-14 01:41:44,151 7f807cc7ae00 DEBUG firewalld.service is not enabled 2026-02-14 01:41:44,151 7f807cc7ae00 DEBUG Not possible to open ports <[9093, 9094]>. firewalld.service is not available 2026-02-14 01:41:44,286 7f42bb1d0e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 01:41:45,289 7f6eb21e3e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:41:45,506 7fdd6761ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'list-networks'] 2026-02-14 01:41:45,662 7ff87de7ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 01:41:45,679 7ff87de7ce00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:41:45,679 7ff87de7ce00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:41:46,439 7f0b70a69e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'inspect-image'] 2026-02-14 01:41:46,474 7f0b70a69e00 DEBUG /usr/bin/podman: stdout edc95c9460f10a90f1e921f9347538ed9be0b1c7aee54201d89be4c7dacb23c7,[quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9] 2026-02-14 01:41:46,594 7f0b70a69e00 DEBUG ceph: stdout ceph version 20.2.0-667-geae49298 (eae49298735c216a37f3957204b0fe058b58e376) tentacle (stable) 2026-02-14 01:41:46,768 7f43ed435e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.io/ceph/grafana:12.3.1', '--timeout', '895', '_orch', 'deploy', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460'] 2026-02-14 01:41:46,895 7f43ed435e00 DEBUG Loaded deploy configuration: {'fsid': '29ce0a10-0945-11f1-b2be-d404e6e7d460', 'name': 'grafana.trial014', 'image': '', 'deploy_arguments': [], 'params': {'tcp_ports': [3000], 'allow_ptrace': True}, 'meta': {'service_name': 'grafana', 'ports': [3000], 'ip': None, 'deployed_by': ['quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9'], 'rank': None, 'rank_generation': None, 'extra_container_args': None, 'extra_entrypoint_args': None}, 'config_blobs': {'files': {'grafana.ini': "# This file is generated by cephadm.\n[users]\n default_theme = light\n[auth.anonymous]\n enabled = true\n org_name = 'Main Org.'\n org_role = 'Viewer'\n[server]\n domain = 'trial014'\n protocol = https\n cert_file = /etc/grafana/certs/cert_file\n cert_key = /etc/grafana/certs/cert_key\n http_port = 3000\n http_addr = \n[snapshots]\n external_enabled = false\n[security]\n disable_initial_admin_creation = true\n cookie_secure = true\n cookie_samesite = none\n allow_embedding = true\n[analytics]\n check_for_updates = false\n reporting_enabled = false\n[plugins]\n check_for_plugin_updates = false\n public_key_retrieval_disabled = true", 'provisioning/datasources/ceph-dashboard.yml': "# This file is generated by cephadm.\napiVersion: 1\n\ndeleteDatasources:\n - name: 'Dashboard1'\n orgId: 1\n\ndatasources:\n - name: 'Dashboard1'\n type: 'prometheus'\n access: 'proxy'\n orgId: 1\n url: 'http://trial014:9095'\n basicAuth: false\n isDefault: true\n editable: false\n\n - name: 'Loki'\n type: 'loki'\n access: 'proxy'\n url: ''\n basicAuth: false\n isDefault: false\n editable: false", 'certs/cert_file': '# generated by cephadm\n-----BEGIN CERTIFICATE-----\nMIICyDCCAbACEQDWFuGbwrFDSaO1W/2Qonk4MA0GCSqGSIb3DQEBDQUAMCIxDTAL\nBgNVBAoMBENlcGgxETAPBgNVBAMMCHRyaWFsMDE0MB4XDTI2MDIxNDAxMzQwN1oX\nDTM2MDIxMjAxMzQwN1owIjENMAsGA1UECgwEQ2VwaDERMA8GA1UEAwwIdHJpYWww\nMTQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDAZFJgUdruhEbDg2KF\nbh+5L7LQQ0IBiFrkI9MmBZ7pVetM7Yl+VyTuJeFQ6wYsQlUywcMPpLOEcafbDh9u\nnO/NRDKX/VhaCY9+deYMgQ+XoGA3a69jdllqJQKum/s5VleUOn8TmZ1rN/Rv1cCL\no0Ip0GKuG9xKJq3PoJlKDZzS/j0fJaTnQTZNGoszD9msA+7lZjZ9WBCAntAI0a0U\nxUCvAJT8YBG4KMSh2qXufWWOunvnmG8a9hDrq/IsVHowlbfgsQ+6tBMQIuo3OElg\n5j/JWlS4erC9tYdxrSSupJcR3LO9PrlUQ+cGDSe4sX3Yr+eACXQoyJUxXBj+CAUr\na0oXAgMBAAEwDQYJKoZIhvcNAQENBQADggEBAMAu3dnDu5qLUhUezg90qIXVP19g\nUeDirCGhpwZtoR2WpkLgh1Ceq8TK0nkHUkg3P/mwKTrI2Q5OD6UPyy40kyyx2t1k\nybbTzxo9Xz6F5jOpOuyoKz18M2U3fGDLxO5lbvlLCbeQez2o+AJhcFIv1M0CxGPB\nkbFj48cvz36cxtqSTmie+yl2UzKERQ++WzBhBUSWb1cntIAlbmCbVloseWGGcrmT\nBQ3qJYW3LgVCnSG5AwcajL0dLAhrhcvERBR2MCcprFQrGZhfxMBzMXoeEoIUXiOW\nZgSbAGEF9PrOAEWmAeZOLY1xqzA5P9Ww6p5I5UY8fTSorfqh4OVSGvNtCmM=\n-----END CERTIFICATE-----\n', 'certs/cert_key': '# generated by cephadm\n-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDAZFJgUdruhEbD\ng2KFbh+5L7LQQ0IBiFrkI9MmBZ7pVetM7Yl+VyTuJeFQ6wYsQlUywcMPpLOEcafb\nDh9unO/NRDKX/VhaCY9+deYMgQ+XoGA3a69jdllqJQKum/s5VleUOn8TmZ1rN/Rv\n1cCLo0Ip0GKuG9xKJq3PoJlKDZzS/j0fJaTnQTZNGoszD9msA+7lZjZ9WBCAntAI\n0a0UxUCvAJT8YBG4KMSh2qXufWWOunvnmG8a9hDrq/IsVHowlbfgsQ+6tBMQIuo3\nOElg5j/JWlS4erC9tYdxrSSupJcR3LO9PrlUQ+cGDSe4sX3Yr+eACXQoyJUxXBj+\nCAUra0oXAgMBAAECggEADotG+EXChtzJu1wCUFNGjf7/cI5fvgfrKcCqOf9GHR5D\n/yPHN3LVCt/NjGAdWQPcDE3o4L+IW/nPllgXTjW09yUxrJsYA8PMl43IH1KJCuMH\n2F4JUJE7fzJfD94g0XEn27/JqCypDC2bCp1JGgVJ1NQOXFp6KReKyjMPg4aglV6b\nG3UZp8LDBLzEzp3Qs5jhL9YCH+ETRr5tkrZFOFJsTSckMGmzCQ5QJKsz5tDrofo9\nyzMIDjkokdSUgc8UHuXDMjWkECWRpfcQoRNVeTQ+ZND0k4HXpJEk2ecqQKF4UK6f\nf75st1wEssyWYqGfUKZcKzfVzgfAHYyF2GV6h3FmSQKBgQD4pk5plfHs2PV1L0TB\nmCMeLOKngIZmy+foM/wT/gaJ/gXSaUJ1V2syZ1/igO2jZZppKMpEApb+vMPRMnRx\nsMymLHHIrwPoQkUgdtM6aGO1IoMFyPMs9gvQLYsZh5YsltulRhoxUKDobjkjujRq\nNCKKtqEP6cTlCUMYnQxqPdsdBQKBgQDGFEbEtAiB/c9ViL9dhUkIE3sahlv/dMtZ\n7tnFDFwx2PNjQ1KezUkGAktVYgxDve4D0uXE7TbYtRdOGZsbQd8KyBAsTstorfcT\nnjCZrp3Pi2cFrOcQzlKKeztNmJOhBV4XURlP26OckElaYNZMuejUxggbvXqGqnNA\nggTXUb3VawKBgQDauOgG5vhtgJfCYjcYGKzjkVY7Q8p2SG2SC5B1QaIc5o/We/MS\n5zgLmXV1+DWR4wFF+bSayLBsGj4iGkDl9UC/+B6LCrDkMM8/9GZwXX0/y8D3aMHI\nThMsJ7PRXDyUsos5wgWuVZ+FddfbfqM2vO+Zl3bsWZVyZ4hW8WcHWawRxQKBgEzA\nOpOpdsp3wEjSTeQ86wl4EvE6MpePd+QHCtaxbV/bQqOpyG/D95GeffY9oab2By3w\nIBvoFqJtuPQIM/Oh2Hzi/HFOrr6G+f/eAdLquDddOrQWRQpcB9uk2Fu+zSVNYSID\nyJ0K7CCUP/gHIcJlHvSjLy0ghEN99F6AgsOz5qy1AoGBAPa9+KxjN5k0Rbq7hSN6\noYd72KPE2hyC1rZfmOJBYn/a5a72tIahX2BNhgyXOmB65yZpYrDaxSsqxc6RFkQK\nFv9qwxpifEQ8/AwTNYkgbkLc2WXB9LT5QCCiuE7XcOyGdGfIxvdoGwke60Rkec6f\n+3ubSlyzhTdJGepyRdWYrXSG\n-----END PRIVATE KEY-----\n', 'provisioning/dashboards/default.yml': "# This file is generated by cephadm.\napiVersion: 1\n\nproviders:\n - name: 'Ceph Dashboard'\n orgId: 1\n folder: ''\n type: file\n disableDeletion: false\n updateIntervalSeconds: 3\n editable: false\n options:\n path: '/etc/grafana/provisioning/dashboards'", '/etc/grafana/provisioning/dashboards/host-details.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.3.2"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n },\n {\n "id": "singlestat",\n "name": "Singlestat",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [\n {\n "asDropdown": true,\n "icon": "external link",\n "includeVars": true,\n "keepTime": true,\n "tags": [ ],\n "targetBlank": false,\n "title": "Browse Dashboards",\n "tooltip": "",\n "type": "dashboards",\n "url": ""\n }\n ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "$ceph_hosts System Overview",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "format": "none",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 5,\n "w": 3,\n "x": 0,\n "y": 1\n },\n "id": 3,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "count(sum by (ceph_daemon) (ceph_osd_metadata{cluster=~\\"$cluster|\\", hostname=~\\"$ceph_hosts\\"}))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "OSDs",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "% Utilization",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "percent"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 10,\n "w": 6,\n "x": 3,\n "y": 1\n },\n "id": 4,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "sum by (mode) (\\n rate(node_cpu{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\", mode=~\\"(irq|nice|softirq|steal|system|user|iowait)\\"}[$__rate_interval]) or\\n rate(node_cpu_seconds_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\", mode=~\\"(irq|nice|softirq|steal|system|user|iowait)\\"}[$__rate_interval])\\n) / (\\n scalar(\\n sum(rate(node_cpu{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\n rate(node_cpu_seconds_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]))\\n ) * 100\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{mode}}",\n "refId": "A"\n }\n ],\n "title": "CPU Utilization",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "RAM used",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "bytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 10,\n "w": 6,\n "x": 9,\n "y": 1\n },\n "id": 5,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "total",\n "color": "#bf1b00",\n "fill": 0,\n "linewidth": 2,\n "stack": false\n }\n ],\n "targets": [\n {\n "expr": "node_memory_MemFree{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\nnode_memory_MemFree_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Free",\n "refId": "A"\n },\n {\n "expr": "node_memory_MemTotal{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\nnode_memory_MemTotal_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "total",\n "refId": "B"\n },\n {\n "expr": "(\\n node_memory_Cached{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\n node_memory_Cached_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n) + (\\n node_memory_Buffers{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\n node_memory_Buffers_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n) + (\\n node_memory_Slab{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\n node_memory_Slab_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "buffers/cache",\n "refId": "C"\n },\n {\n "expr": "(\\n node_memory_MemTotal{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\n node_memory_MemTotal_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n) - (\\n (\\n node_memory_MemFree{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\n node_memory_MemFree_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n ) + (\\n node_memory_Cached{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\n node_memory_Cached_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n ) + (\\n node_memory_Buffers{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\n node_memory_Buffers_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n ) + (\\n node_memory_Slab{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"} or\\n node_memory_Slab_bytes{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}\\n )\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "used",\n "refId": "D"\n }\n ],\n "title": "RAM Usage",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Send (-) / Receive (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "decbytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 10,\n "w": 6,\n "x": 15,\n "y": 1\n },\n "id": 6,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*tx/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "sum by (device) (\\n rate(node_network_receive_bytes{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\",device!=\\"lo\\"}[$__rate_interval]) or\\n rate(node_network_receive_bytes_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\",device!=\\"lo\\"}[$__rate_interval])\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}.rx",\n "refId": "A"\n },\n {\n "expr": "sum by (device) (\\n rate(node_network_transmit_bytes{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\",device!=\\"lo\\"}[$__rate_interval]) or\\n rate(node_network_transmit_bytes_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\",device!=\\"lo\\"}[$__rate_interval])\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}.tx",\n "refId": "B"\n }\n ],\n "title": "Network Load",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Send (-) / Receive (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "pps"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 5,\n "w": 3,\n "x": 21,\n "y": 1\n },\n "id": 7,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*tx/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "rate(node_network_receive_drop{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\nrate(node_network_receive_drop_total{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}.rx",\n "refId": "A"\n },\n {\n "expr": "rate(node_network_transmit_drop{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\nrate(node_network_transmit_drop_total{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}.tx",\n "refId": "B"\n }\n ],\n "title": "Network drop rate",\n "type": "timeseries"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Each OSD consists of a Journal/WAL partition and a data partition. The RAW Capacity shown is the sum of the data partitions across all OSDs on the selected OSD hosts.",\n "format": "bytes",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 5,\n "w": 3,\n "x": 0,\n "y": 6\n },\n "id": 8,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum(\\n ceph_osd_stat_bytes{cluster=~\\"$cluster|\\", } and\\n on (ceph_daemon) ceph_disk_occupation{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\", cluster=~\\"$cluster|\\", }\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Raw Capacity",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Send (-) / Receive (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "pps"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 5,\n "w": 3,\n "x": 21,\n "y": 6\n },\n "id": 9,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*tx/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "rate(node_network_receive_errs{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\nrate(node_network_receive_errs_total{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}.rx",\n "refId": "A"\n },\n {\n "expr": "rate(node_network_transmit_errs{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\nrate(node_network_transmit_errs_total{instance=~\\"$ceph_hosts([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}.tx",\n "refId": "B"\n }\n ],\n "title": "Network error rate",\n "type": "timeseries"\n },\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 11\n },\n "id": 10,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "OSD Disk Performance Statistics",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "ops"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 11,\n "x": 0,\n "y": 12\n },\n "id": 11,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*reads/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "label_replace(\\n (\\n rate(node_disk_writes_completed{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\n rate(node_disk_writes_completed_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) * on(instance, device) group_left(ceph_daemon) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{cluster=~\\"$cluster|\\", }, \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}({{ceph_daemon}}) writes",\n "refId": "A"\n },\n {\n "expr": "label_replace(\\n (\\n rate(node_disk_reads_completed{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\n rate(node_disk_reads_completed_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) * on(instance, device) group_left(ceph_daemon) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{cluster=~\\"$cluster|\\", }, \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}({{ceph_daemon}}) reads",\n "refId": "B"\n }\n ],\n "title": "$ceph_hosts Disk IOPS",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "Bps"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 11,\n "x": 12,\n "y": 12\n },\n "id": 12,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*read/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "label_replace(\\n (\\n rate(node_disk_bytes_written{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\n rate(node_disk_written_bytes_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) * on(instance, device)\\ngroup_left(ceph_daemon) label_replace(\\n label_replace(ceph_disk_occupation_human{cluster=~\\"$cluster|\\", }, \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}({{ceph_daemon}}) write",\n "refId": "A"\n },\n {\n "expr": "label_replace(\\n (\\n rate(node_disk_bytes_read{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) or\\n rate(node_disk_read_bytes_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) * on(instance, device)\\ngroup_left(ceph_daemon) label_replace(\\n label_replace(ceph_disk_occupation_human{cluster=~\\"$cluster|\\", }, \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}({{ceph_daemon}}) read",\n "refId": "B"\n }\n ],\n "title": "$ceph_hosts Throughput by Disk",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "s"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 11,\n "x": 0,\n "y": 21\n },\n "id": 13,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "max by(instance, device) (\\n label_replace(\\n (\\n (rate(node_disk_write_time_seconds_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])) /\\n clamp_min(rate(node_disk_writes_completed_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]), 0.001) or\\n (rate(node_disk_read_time_seconds_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval])) /\\n clamp_min(rate(node_disk_reads_completed_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]), 0.001)\\n ),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n )\\n) * on(instance, device) group_left(ceph_daemon) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"},\\n \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}({{ceph_daemon}})",\n "refId": "A"\n }\n ],\n "title": "$ceph_hosts Disk Latency",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "%Util",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "percent"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 11,\n "x": 12,\n "y": 21\n },\n "id": 14,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "label_replace(\\n (\\n (rate(node_disk_io_time_ms{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) / 10) or\\n rate(node_disk_io_time_seconds_total{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\"}[$__rate_interval]) * 100\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) * on(instance, device) group_left(ceph_daemon) label_replace(\\n label_replace(ceph_disk_occupation_human{instance=~\\"($ceph_hosts)([\\\\\\\\\\\\\\\\.:].*)?\\", cluster=~\\"$cluster|\\", },\\n \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}}({{ceph_daemon}})",\n "refId": "A"\n }\n ],\n "title": "$ceph_hosts Disk utilization",\n "type": "timeseries"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "null",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "instance"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Instance"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Slow Ops"\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 0,\n "y": 30\n },\n "id": 15,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "10.4.0",\n "styles": [ ],\n "targets": [\n {\n "expr": "topk(10,\\n (sum by (instance)(ceph_daemon_health_metrics{type=\\"SLOW_OPS\\", ceph_daemon=~\\"osd.*\\", cluster=~\\"$cluster|\\", }))\\n)\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Top Slow Ops per Host",\n "transformations": [\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time": true,\n "cluster": true\n },\n "includeByName": { },\n "indexByName": { },\n "renameByName": { }\n }\n }\n ],\n "type": "table"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin",\n "overview"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": null,\n "multi": false,\n "name": "ceph_hosts",\n "options": [ ],\n "query": "label_values({__name__=~\\"ceph_.+_metadata\\", cluster=~\\"$cluster|\\", }, hostname)",\n "refresh": 1,\n "regex": "([^.]*).*",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "Host Details",\n "uid": "rtOg0AiWz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/multi-cluster-overview.json': '{\n "__inputs": [ ],\n "__requires": [ ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [\n {\n "asDropdown": true,\n "icon": "external link",\n "includeVars": true,\n "keepTime": true,\n "tags": [ ],\n "targetBlank": false,\n "title": "Browse Dashboards",\n "tooltip": "",\n "type": "dashboards",\n "url": ""\n }\n ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 1\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "Clusters",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "text",\n "value": null\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "Warning"\n },\n "properties": [\n {\n "id": "thresholds",\n "value": {\n "mode": "absolute",\n "steps": [\n {\n "color": "text",\n "value": null\n },\n {\n "color": "semi-dark-yellow",\n "value": 1\n }\n ]\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Error"\n },\n "properties": [\n {\n "id": "thresholds",\n "value": {\n "mode": "absolute",\n "steps": [\n {\n "color": "text",\n "value": null\n },\n {\n "color": "semi-dark-red",\n "value": 1\n }\n ]\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Healthy"\n },\n "properties": [\n {\n "id": "thresholds",\n "value": {\n "mode": "absolute",\n "steps": [\n {\n "color": "text",\n "value": null\n },\n {\n "color": "semi-dark-green",\n "value": 1\n }\n ]\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 7,\n "w": 5,\n "x": 0,\n "y": 2\n },\n "id": 3,\n "links": [ ],\n "options": {\n "colorMode": "value",\n "graphMode": "none",\n "justifyMode": "center",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "count(ceph_health_status==0) or vector(0)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Healthy",\n "refId": "A"\n },\n {\n "datasource": "$datasource",\n "expr": "count(ceph_health_status==1)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Warning",\n "refId": "B"\n },\n {\n "datasource": "$datasource",\n "expr": "count(ceph_health_status==2)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Error",\n "refId": "C"\n }\n ],\n "title": "Status",\n "transparent": false,\n "type": "stat"\n },\n {\n "columns": [ ],\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "left",\n "cellOptions": {\n "type": "color-text"\n },\n "filterable": false,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "text"\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "Value #A"\n },\n "properties": [\n {\n "id": "mappings",\n "value": [\n {\n "options": {\n "0": {\n "color": "semi-dark-green",\n "index": 2,\n "text": "Healthy"\n },\n "1": {\n "color": "semi-dark-yellow",\n "index": 0,\n "text": "Warning"\n },\n "2": {\n "color": "semi-dark-red",\n "index": 1,\n "text": "Error"\n }\n },\n "type": "value"\n }\n ]\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Capacity Used"\n },\n "properties": [\n {\n "id": "unit",\n "value": "bytes"\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Cluster"\n },\n "properties": [\n {\n "id": "links",\n "value": [\n {\n "title": "",\n "url": "/d/edtb0oxdq/ceph-cluster?var-cluster=${__data.fields.Cluster}&${DS_PROMETHEUS:queryparam}"\n }\n ]\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Alerts"\n },\n "properties": [\n {\n "id": "mappings",\n "value": [\n {\n "options": {\n "match": null,\n "result": {\n "index": 0,\n "text": "0"\n }\n },\n "type": "special"\n }\n ]\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 7,\n "w": 19,\n "x": 5,\n "y": 2\n },\n "id": 4,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "9.4.7",\n "styles": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "$datasource"\n },\n "exemplar": false,\n "expr": "ceph_health_status",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "$datasource"\n },\n "exemplar": false,\n "expr": "ceph_mgr_metadata",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "$datasource"\n },\n "exemplar": false,\n "expr": "count(ALERTS{alertstate=\\"firing\\", cluster=~\\"$cluster|\\"})",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "$datasource"\n },\n "exemplar": false,\n "expr": "ceph_cluster_by_class_total_used_bytes",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "D"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Details",\n "transformations": [\n {\n "id": "joinByField",\n "options": {\n "byField": "cluster",\n "mode": "outer"\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time 1": true,\n "Time 2": true,\n "Time 3": true,\n "Time 4": true,\n "Time 5": true,\n "Time 6": true,\n "Value #B": true,\n "__name__ 1": true,\n "__name__ 2": true,\n "__name__ 3": true,\n "ceph_daemon": true,\n "device_class": true,\n "hostname": true,\n "instance 1": true,\n "instance 2": true,\n "instance 3": true,\n "job 1": true,\n "job 2": true,\n "job 3": true,\n "replica 1": true,\n "replica 2": true,\n "replica 3": true\n },\n "indexByName": {\n "Time 1": 8,\n "Time 2": 13,\n "Time 3": 21,\n "Time 4": 7,\n "Time 5": 22,\n "Time 6": 23,\n "Value #A": 1,\n "Value #B": 20,\n "Value #C": 3,\n "Value #D": 6,\n "__name__ 1": 9,\n "__name__ 2": 14,\n "__name__ 3": 24,\n "ceph_daemon": 15,\n "ceph_version": 2,\n "cluster": 0,\n "device_class": 25,\n "hostname": 16,\n "instance 1": 10,\n "instance 2": 17,\n "instance 3": 26,\n "job 1": 11,\n "job 2": 18,\n "job 3": 27,\n "replica 1": 12,\n "replica 2": 19,\n "replica 3": 28\n },\n "renameByName": {\n "Value #A": "Status",\n "Value #C": "Alerts",\n "Value #D": "Capacity Used",\n "ceph_version": "Version",\n "cluster": "Cluster"\n }\n }\n }\n ],\n "type": "table"\n },\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 9\n },\n "id": 5,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "Overview",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "text",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "none"\n }\n },\n "gridPos": {\n "h": 4,\n "w": 3,\n "x": 0,\n "y": 10\n },\n "id": 6,\n "links": [ ],\n "options": {\n "colorMode": "value",\n "graphMode": "none",\n "justifyMode": "center",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "$datasource"\n },\n "exemplar": false,\n "expr": "count(ceph_health_status{cluster=~\\"$cluster|\\"}) or vector(0)",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Cluster Count",\n "transparent": false,\n "type": "stat"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "links": [ ],\n "mappings": [ ],\n "max": 1,\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "semi-dark-yellow",\n "value": 0.75\n },\n {\n "color": "red",\n "value": 0.84999999999999998\n }\n ]\n },\n "unit": "percentunit"\n }\n },\n "gridPos": {\n "h": 8,\n "w": 4,\n "x": 3,\n "y": 10\n },\n "id": 7,\n "interval": "1m",\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "showThresholdLabels": false,\n "showThresholdMarkers": true\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_cluster_total_used_bytes{cluster=~\\"$cluster|\\"}) / sum(ceph_cluster_total_bytes{cluster=~\\"$cluster|\\"})",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Used",\n "refId": "A"\n }\n ],\n "title": "Capacity Used",\n "transparent": false,\n "type": "gauge"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "bytes"\n }\n },\n "gridPos": {\n "h": 4,\n "w": 3,\n "x": 7,\n "y": 10\n },\n "id": 8,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "$datasource"\n },\n "exemplar": false,\n "expr": "sum(ceph_cluster_total_bytes{cluster=~\\"$cluster|\\"})",\n "format": "table",\n "hide": false,\n "instant": false,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Total Capacity",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "none"\n }\n },\n "gridPos": {\n "h": 4,\n "w": 3,\n "x": 10,\n "y": 10\n },\n "id": 9,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "$datasource"\n },\n "exemplar": false,\n "expr": "count(ceph_osd_metadata{cluster=~\\"$cluster|\\"})",\n "format": "table",\n "hide": false,\n "instant": false,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "OSDs",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "none"\n }\n },\n "gridPos": {\n "h": 4,\n "w": 3,\n "x": 13,\n "y": 10\n },\n "id": 10,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "$datasource"\n },\n "exemplar": false,\n "expr": "count(sum by (hostname) (ceph_osd_metadata{cluster=~\\"$cluster|\\"}))",\n "format": "table",\n "hide": false,\n "instant": false,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Hosts",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "ops"\n }\n },\n "gridPos": {\n "h": 4,\n "w": 4,\n "x": 16,\n "y": 10\n },\n "id": 11,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "center",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "$datasource"\n },\n "exemplar": false,\n "expr": "sum(rate(ceph_pool_wr{cluster=~\\"$cluster|\\"}[$__rate_interval]))",\n "format": "time_series",\n "hide": false,\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "Write",\n "range": true,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "$datasource"\n },\n "exemplar": false,\n "expr": "sum(rate(ceph_pool_rd{cluster=~\\"$cluster|\\"}[$__rate_interval]))",\n "format": "time_series",\n "hide": false,\n "intervalFactor": 1,\n "legendFormat": "Read",\n "range": true,\n "refId": "B"\n }\n ],\n "title": "Client IOPS",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "ms"\n }\n },\n "gridPos": {\n "h": 4,\n "w": 4,\n "x": 20,\n "y": 10\n },\n "id": 12,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "center",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "$datasource"\n },\n "exemplar": false,\n "expr": "avg(ceph_osd_apply_latency_ms{cluster=~\\"$cluster|\\"})",\n "format": "time_series",\n "hide": false,\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "Apply",\n "range": true,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "$datasource"\n },\n "exemplar": false,\n "expr": "avg(ceph_osd_commit_latency_ms{cluster=~\\"$cluster|\\"})",\n "format": "time_series",\n "hide": false,\n "intervalFactor": 1,\n "legendFormat": "Commit",\n "range": true,\n "refId": "B"\n }\n ],\n "title": "OSD Latencies",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "text",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "none"\n }\n },\n "gridPos": {\n "h": 4,\n "w": 3,\n "x": 0,\n "y": 14\n },\n "id": 13,\n "links": [ ],\n "options": {\n "colorMode": "value",\n "graphMode": "none",\n "justifyMode": "center",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "$datasource"\n },\n "exemplar": false,\n "expr": "count(ALERTS{alertstate=\\"firing\\", cluster=~\\"$cluster|\\"}) or vector(0)",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Alert Count",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "bytes"\n }\n },\n "gridPos": {\n "h": 4,\n "w": 3,\n "x": 7,\n "y": 14\n },\n "id": 14,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "$datasource"\n },\n "exemplar": false,\n "expr": "sum(ceph_cluster_total_used_bytes{cluster=~\\"$cluster|\\"})",\n "format": "table",\n "hide": false,\n "instant": false,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Total Used",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "s"\n }\n },\n "gridPos": {\n "h": 4,\n "w": 3,\n "x": 10,\n "y": 14\n },\n "id": 15,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "$datasource"\n },\n "exemplar": false,\n "expr": "predict_linear(avg(increase(ceph_cluster_total_used_bytes{cluster=~\\"${Cluster}\\"}[1d]))[7d:1h],120)",\n "format": "time_series",\n "hide": false,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Capacity Prediction",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "none"\n }\n },\n "gridPos": {\n "h": 4,\n "w": 3,\n "x": 13,\n "y": 14\n },\n "id": 16,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "$datasource"\n },\n "exemplar": false,\n "expr": "count(ceph_pool_metadata{cluster=~\\"$cluster|\\"})",\n "format": "table",\n "hide": false,\n "instant": false,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Pools",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "binBps"\n }\n },\n "gridPos": {\n "h": 4,\n "w": 4,\n "x": 16,\n "y": 14\n },\n "id": 17,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "center",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "$datasource"\n },\n "exemplar": false,\n "expr": "sum(rate(ceph_pool_rd_bytes{cluster=~\\"$cluster|\\"}[$__rate_interval]))",\n "format": "time_series",\n "hide": false,\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "Write",\n "range": true,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "$datasource"\n },\n "exemplar": false,\n "expr": "sum(rate(ceph_pool_wr_bytes{cluster=~\\"$cluster|\\"}[$__rate_interval]))",\n "format": "time_series",\n "hide": false,\n "intervalFactor": 1,\n "legendFormat": "Read",\n "range": true,\n "refId": "B"\n }\n ],\n "title": "Client Bandwidth",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "binBps"\n }\n },\n "gridPos": {\n "h": 4,\n "w": 4,\n "x": 20,\n "y": 14\n },\n "id": 18,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "center",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "$datasource"\n },\n "exemplar": false,\n "expr": "sum(rate(ceph_osd_recovery_ops{cluster=~\\"$cluster|\\"}[$__rate_interval]))",\n "format": "time_series",\n "hide": false,\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "Write",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Recovery Rate",\n "transparent": false,\n "type": "stat"\n },\n {\n "collapse": false,\n "collapsed": true,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 18\n },\n "id": 19,\n "panels": [\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "text",\n "value": null\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "Critical"\n },\n "properties": [\n {\n "id": "thresholds",\n "value": {\n "mode": "absolute",\n "steps": [\n {\n "color": "text",\n "value": null\n },\n {\n "color": "semi-dark-red",\n "value": 1\n }\n ]\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Warning"\n },\n "properties": [\n {\n "id": "thresholds",\n "value": {\n "mode": "absolute",\n "steps": [\n {\n "color": "text",\n "value": null\n },\n {\n "color": "semi-dark-yellow",\n "value": 1\n }\n ]\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 7,\n "w": 5,\n "x": 0,\n "y": 19\n },\n "id": 20,\n "links": [ ],\n "options": {\n "colorMode": "value",\n "graphMode": "area",\n "justifyMode": "center",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "count(ALERTS{alertstate=\\"firing\\",severity=\\"critical\\", cluster=~\\"$cluster|\\"}) OR vector(0)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Critical",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": "$datasource",\n "expr": "count(ALERTS{alertstate=\\"firing\\",severity=\\"warning\\", cluster=~\\"$cluster|\\"}) OR vector(0)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Warning",\n "range": false,\n "refId": "B"\n }\n ],\n "title": "Status",\n "transparent": false,\n "type": "stat"\n },\n {\n "columns": [ ],\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "auto",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 19,\n "x": 5,\n "y": 19\n },\n "id": 21,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true,\n "sortBy": [\n {\n "desc": false,\n "displayName": "Severity"\n }\n ]\n },\n "pluginVersion": "9.4.7",\n "styles": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "$datasource"\n },\n "exemplar": false,\n "expr": "ALERTS{alertstate=\\"firing\\", cluster=~\\"$cluster|\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Alerts",\n "transformations": [\n {\n "id": "joinByField",\n "options": {\n "byField": "cluster",\n "mode": "outer"\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time": true,\n "Value": true,\n "__name__": true,\n "instance": true,\n "job": true,\n "oid": true,\n "replica": true,\n "type": true\n },\n "indexByName": {\n "Time": 0,\n "Value": 9,\n "__name__": 1,\n "alertname": 2,\n "alertstate": 4,\n "cluster": 3,\n "instance": 6,\n "job": 7,\n "severity": 5,\n "type": 8\n },\n "renameByName": {\n "alertname": "Name",\n "alertstate": "State",\n "cluster": "Cluster",\n "severity": "Severity"\n }\n }\n }\n ],\n "type": "table"\n },\n {\n "datasource": {\n "type": "datasource",\n "uid": "grafana"\n },\n "gridPos": {\n "h": 8,\n "w": 24,\n "x": 0,\n "y": 26\n },\n "id": 22,\n "limit": 10,\n "onlyAlertsOnDashboard": true,\n "options": {\n "alertName": "",\n "dashboardAlerts": false,\n "groupBy": [ ],\n "groupMode": "default",\n "maxItems": 20,\n "sortOrder": 1,\n "stateFilter": {\n "error": true,\n "firing": true,\n "noData": false,\n "normal": false,\n "pending": true\n },\n "viewMode": "list"\n },\n "show": "current",\n "sortOrder": 1,\n "stateFilter": [ ],\n "title": "Alerts(Grouped)",\n "type": "alertlist"\n }\n ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "Alerts",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "collapse": false,\n "collapsed": true,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 29\n },\n "id": 23,\n "panels": [\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "percentage",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "percentunit"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 0,\n "y": 30\n },\n "id": 24,\n "options": {\n "legend": {\n "calcs": [\n "last"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "topk(5, ceph_cluster_total_used_bytes/ceph_cluster_total_bytes)",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{cluster}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Top 5 - Capacity Utilization(%)",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "percentage",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "ops"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 8,\n "y": 30\n },\n "id": 25,\n "options": {\n "legend": {\n "calcs": [\n "last"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "topk(10, sum by (cluster) (rate(ceph_osd_op_w[$__rate_interval])) \\n+ sum by (cluster) (rate(ceph_osd_op_r[$__rate_interval])) )",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{cluster}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Top 5 - Cluster IOPS",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "percentunit"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 16,\n "y": 30\n },\n "id": 26,\n "options": {\n "legend": {\n "calcs": [\n "last"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "topk(10, ceph_pool_bytes_used{cluster=~\\"$cluster|\\", }/ceph_pool_max_avail{cluster=~\\"$cluster|\\", } * on(pool_id, cluster) group_left(instance, name) ceph_pool_metadata{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{cluster}} - {{name}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Top 10 - Capacity Utilization(%) by Pool",\n "type": "timeseries"\n }\n ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "Cluster Stats",\n "titleSize": "h6",\n "type": "row"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 22,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": ".*",\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": "cluster",\n "multi": true,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "Ceph - Multi-cluster",\n "uid": "BnxelG7Sx",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/ceph-nvmeof-performance.json': '{\n "__inputs": [ ],\n "__requires": [ ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "Ceph NVMe-oF gateways overview",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [\n {\n "asDropdown": true,\n "icon": "external link",\n "includeVars": true,\n "keepTime": true,\n "tags": [ ],\n "targetBlank": false,\n "title": "Browse Dashboards",\n "tooltip": "",\n "type": "dashboards",\n "url": ""\n }\n ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "Performance",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "percentunit"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 0,\n "y": 1\n },\n "id": 3,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "maxHeight": 600,\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "avg by(instance) (rate(ceph_nvmeof_reactor_seconds_total{mode=\'busy\',instance=~\'$gateway\'}[$__rate_interval]))",\n "format": "",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "AVG Reactor CPU Usage by Gateway",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "percentunit"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 8,\n "y": 1\n },\n "id": 4,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "maxHeight": 600,\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "avg by (instance) (rate(ceph_nvmeof_reactor_seconds_total{mode=\'busy\', instance=~\'$gateway.*\'}[$__rate_interval]))",\n "format": "",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Reactor Threads CPU Usage : $gateway",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "s"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 16,\n "y": 1\n },\n "id": 5,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "maxHeight": 600,\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "avg((rate(ceph_nvmeof_bdev_read_seconds_total{instance=~\'$gateway\'}[$__rate_interval]) / rate(ceph_nvmeof_bdev_reads_completed_total{instance=~\'$gateway\'}[$__rate_interval])) > 0)",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "Reads",\n "range": true,\n "refId": "A"\n },\n {\n "datasource": "$datasource",\n "expr": "avg((rate(ceph_nvmeof_bdev_write_seconds_total{instance=~\'$gateway\'}[$__rate_interval]) / rate(ceph_nvmeof_bdev_writes_completed_total{instance=~\'$gateway\'}[$__rate_interval])) > 0)",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "Writes",\n "range": true,\n "refId": "B"\n }\n ],\n "title": "AVG I/O Latency",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "locale"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 0,\n "y": 9\n },\n "id": 6,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "maxHeight": 600,\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum by(instance) (rate(ceph_nvmeof_bdev_reads_completed_total{instance=~\'$gateway\'}[$__rate_interval]) + rate(ceph_nvmeof_bdev_writes_completed_total{instance=~\'$gateway\'}[$__rate_interval]))",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "IOPS by Gateway",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "locale"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 8,\n "y": 9\n },\n "id": 7,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "maxHeight": 600,\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum by(nqn) ((rate(ceph_nvmeof_bdev_reads_completed_total{instance=~\'$gateway\'}[$__rate_interval]) + rate(ceph_nvmeof_bdev_writes_completed_total{instance=~\'$gateway\'}[$__rate_interval])) * on(instance,bdev_name) group_right ceph_nvmeof_subsystem_namespace_metadata{instance=~\'$gateway\'})",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "IOPS by NVMe-oF Subsystem",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "locale"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 16,\n "y": 9\n },\n "id": 8,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "maxHeight": 600,\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "topk(5, (sum by(pool_name, rbd_name) (((rate(ceph_nvmeof_bdev_reads_completed_total{instance=~\'$gateway\'}[$__rate_interval]) + rate(ceph_nvmeof_bdev_writes_completed_total{instance=~\'$gateway\'}[$__rate_interval])) * on(instance,bdev_name) group_right ceph_nvmeof_bdev_metadata{instance=~\'$gateway\'}) * on(instance, bdev_name) group_left(nqn) ceph_nvmeof_subsystem_namespace_metadata{nqn=~\'$subsystem\',instance=~\'$gateway\'})))",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{pool_name}}/{{rbd_name}}",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "TOP 5 - IOPS by device for $subsystem",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "binBps"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 0,\n "y": 17\n },\n "id": 9,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "maxHeight": 600,\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum by(instance) (rate(ceph_nvmeof_bdev_read_bytes_total{instance=~\'$gateway\'}[$__rate_interval]) + rate(ceph_nvmeof_bdev_written_bytes_total{instance=~\'$gateway\'}[$__rate_interval]))",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Throughput by Gateway",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "binBps"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 8,\n "y": 17\n },\n "id": 10,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "maxHeight": 600,\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum by(nqn) ((rate(ceph_nvmeof_bdev_read_bytes_total{instance=~\'$gateway\'}[$__rate_interval]) + rate(ceph_nvmeof_bdev_written_bytes_total{instance=~\'$gateway\'}[$__rate_interval])) * on(instance,bdev_name) group_right ceph_nvmeof_subsystem_namespace_metadata{instance=~\'$gateway\'})",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Throughput by NVMe-oF Subsystem",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "binBps"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 16,\n "y": 17\n },\n "id": 11,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "maxHeight": 600,\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "topk(5, (sum by(pool_name, rbd_name) (((rate(ceph_nvmeof_bdev_read_bytes_total{instance=~\'$gateway\'}[$__rate_interval]) + rate(ceph_nvmeof_bdev_written_bytes_total{instance=~\'$gateway\'}[$__rate_interval])) * on(instance,bdev_name) group_right ceph_nvmeof_bdev_metadata{instance=~\'$gateway\'}) * on(instance, bdev_name) group_left(nqn) ceph_nvmeof_subsystem_namespace_metadata{nqn=~\'$subsystem\',instance=~\'$gateway\'})))",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "TOP 5 - Throughput by device for $subsystem",\n "type": "timeseries"\n }\n ],\n "refresh": "10s",\n "rows": [ ],\n "schemaVersion": "39",\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": {\n "text": {\n "selected": false,\n "text": "840834cc-05a3-11f0-baba-0200229b9601",\n "value": "840834cc-05a3-11f0-baba-0200229b9601"\n },\n "value": {\n "selected": false,\n "text": "840834cc-05a3-11f0-baba-0200229b9601",\n "value": "840834cc-05a3-11f0-baba-0200229b9601"\n }\n },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": {\n "text": {\n "selected": false,\n "text": "All",\n "value": "$__all"\n },\n "value": {\n "selected": false,\n "text": "All",\n "value": "$__all"\n }\n },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": "Gateway Group",\n "multi": false,\n "name": "group",\n "options": [ ],\n "query": "label_values(ceph_nvmeof_gateway_info,group)",\n "refresh": 2,\n "regex": "",\n "sort": 0,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": {\n "text": {\n "selected": false,\n "text": "All",\n "value": "$__all"\n },\n "value": {\n "selected": false,\n "text": "All",\n "value": "$__all"\n }\n },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": "Gateway Hostname",\n "multi": false,\n "name": "gateway",\n "options": [ ],\n "query": "label_values(ceph_nvmeof_gateway_info{group=~\'$group\'},hostname)",\n "refresh": 2,\n "regex": "",\n "sort": 0,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": {\n "text": {\n "selected": false,\n "text": "All",\n "value": "$__all"\n },\n "value": {\n "selected": false,\n "text": "All",\n "value": "$__all"\n }\n },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": "Subsystem NQN",\n "multi": false,\n "name": "subsystem",\n "options": [ ],\n "query": "label_values(ceph_nvmeof_subsystem_metadata{group=~\'$group\'},nqn)",\n "refresh": 1,\n "regex": "",\n "sort": 0,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-15m",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "Ceph NVMe-oF Gateways - Performance",\n "uid": "feeuv1dno43r4deed",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/radosgw-detail.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.0.0"\n },\n {\n "id": "grafana-piechart-panel",\n "name": "Pie Chart",\n "type": "panel",\n "version": "1.3.3"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [\n {\n "asDropdown": true,\n "icon": "external link",\n "includeVars": true,\n "keepTime": true,\n "tags": [ ],\n "targetBlank": false,\n "title": "Browse Dashboards",\n "tooltip": "",\n "type": "dashboards",\n "url": ""\n }\n ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "RGW Host Detail : $rgw_servers",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "s"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 0,\n "y": 1\n },\n "id": 3,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum by (instance_id) (\\n rate(ceph_rgw_op_get_obj_lat_sum{cluster=~\\"$cluster|\\", }[$__rate_interval]) /\\n rate(ceph_rgw_op_get_obj_lat_count{cluster=~\\"$cluster|\\", }[$__rate_interval])\\n) * on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "GET {{ceph_daemon}}",\n "refId": "A"\n },\n {\n "expr": "sum by (instance_id) (\\n rate(ceph_rgw_op_put_obj_lat_sum{cluster=~\\"$cluster|\\", }[$__rate_interval]) /\\n rate(ceph_rgw_op_put_obj_lat_count{cluster=~\\"$cluster|\\", }[$__rate_interval])\\n) * on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "PUT {{ceph_daemon}}",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "$rgw_servers GET/PUT Latencies",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "s",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "bytes"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 8,\n "w": 7,\n "x": 6,\n "y": 1\n },\n "id": 4,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "rate(ceph_rgw_op_get_obj_bytes{cluster=~\\"$cluster|\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "GETs {{ceph_daemon}}",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_rgw_op_put_obj_bytes{cluster=~\\"$cluster|\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon)\\n ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "PUTs {{ceph_daemon}}",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Bandwidth by HTTP Operation",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "bytes",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": {\n "GETs": "#7eb26d",\n "Other": "#447ebc",\n "PUTs": "#eab839",\n "Requests": "#3f2b5b",\n "Requests Failed": "#bf1b00"\n },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 8,\n "w": 7,\n "x": 13,\n "y": 1\n },\n "id": 5,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "rate(ceph_rgw_failed_req{cluster=~\\"$cluster|\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Requests Failed {{ceph_daemon}}",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_rgw_get{cluster=~\\"$cluster|\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "GETs {{ceph_daemon}}",\n "refId": "B"\n },\n {\n "expr": "rate(ceph_rgw_put{cluster=~\\"$cluster|\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "PUTs {{ceph_daemon}}",\n "refId": "C"\n },\n {\n "expr": "(\\n rate(ceph_rgw_req{cluster=~\\"$cluster|\\", }[$__rate_interval]) -\\n (\\n rate(ceph_rgw_get{cluster=~\\"$cluster|\\", }[$__rate_interval]) +\\n rate(ceph_rgw_put{cluster=~\\"$cluster|\\", }[$__rate_interval])\\n )\\n) * on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Other {{ceph_daemon}}",\n "refId": "D"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "HTTP Request Breakdown",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n }\n },\n "mappings": [ ]\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "Failures"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "#bf1b00",\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "GETs"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "#7eb26d",\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Other (HEAD,POST,DELETE)"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "#447ebc",\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "PUTs"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "#eab839",\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Requests"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "#3f2b5b",\n "mode": "fixed"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 4,\n "x": 20,\n "y": 1\n },\n "id": 6,\n "options": {\n "displayLabels": [ ],\n "legend": {\n "calcs": [ ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "values": [\n "percent",\n "value"\n ]\n },\n "pieType": "pie",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "tooltip": {\n "mode": "single",\n "sort": "none"\n }\n },\n "targets": [\n {\n "expr": "rate(ceph_rgw_failed_req{cluster=~\\"$cluster|\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Failures {{ceph_daemon}}",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_rgw_get{cluster=~\\"$cluster|\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "GETs {{ceph_daemon}}",\n "refId": "B"\n },\n {\n "expr": "rate(ceph_rgw_put{cluster=~\\"$cluster|\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "PUTs {{ceph_daemon}}",\n "refId": "C"\n },\n {\n "expr": "(\\n rate(ceph_rgw_req{cluster=~\\"$cluster|\\", }[$__rate_interval]) -\\n (\\n rate(ceph_rgw_get{cluster=~\\"$cluster|\\", }[$__rate_interval]) +\\n rate(ceph_rgw_put{cluster=~\\"$cluster|\\", }[$__rate_interval])\\n )\\n) * on (instance_id) group_left (ceph_daemon)\\n ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Other (DELETE,LIST) {{ceph_daemon}}",\n "refId": "D"\n }\n ],\n "title": "Workload Breakdown",\n "type": "piechart"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin",\n "overview"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": "",\n "multi": false,\n "name": "rgw_servers",\n "options": [ ],\n "query": "label_values(ceph_rgw_metadata{cluster=~\\"$cluster|\\", }, ceph_daemon)",\n "refresh": 1,\n "regex": "",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "RGW Instance Detail",\n "uid": "x5ARzZtmk",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/radosgw-overview.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.0.0"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [\n {\n "asDropdown": true,\n "icon": "external link",\n "includeVars": true,\n "keepTime": true,\n "tags": [ ],\n "targetBlank": false,\n "title": "Browse Dashboards",\n "tooltip": "",\n "type": "dashboards",\n "url": ""\n }\n ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "RGW Overview - All Gateways",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "s"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 0,\n "y": 1\n },\n "id": 3,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "label_replace(\\n rate(ceph_rgw_op_get_obj_lat_sum{cluster=~\\"$cluster|\\", }[$__rate_interval]) /\\n rate(ceph_rgw_op_get_obj_lat_count{cluster=~\\"$cluster|\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", },\\n \\"rgw_host\\", \\"$1\\", \\"ceph_daemon\\", \\"rgw.(.*)\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "GET {{rgw_host}}",\n "refId": "A"\n },\n {\n "expr": "label_replace(\\n rate(ceph_rgw_op_put_obj_lat_sum{cluster=~\\"$cluster|\\", }[$__rate_interval]) /\\n rate(ceph_rgw_op_put_obj_lat_count{cluster=~\\"$cluster|\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", },\\n \\"rgw_host\\", \\"$1\\", \\"ceph_daemon\\", \\"rgw.(.*)\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "PUT {{rgw_host}}",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Average GET/PUT Latencies by RGW Instance",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "s",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "none"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 7,\n "x": 8,\n "y": 1\n },\n "id": 4,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum by (rgw_host) (\\n label_replace(\\n rate(ceph_rgw_req{cluster=~\\"$cluster|\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", },\\n \\"rgw_host\\", \\"$1\\", \\"ceph_daemon\\", \\"rgw.(.*)\\"\\n )\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{rgw_host}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Total Requests/sec by RGW Instance",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "none",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "Latencies are shown stacked, without a yaxis to provide a visual indication of GET latency imbalance across RGW hosts",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "s"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 6,\n "x": 15,\n "y": 1\n },\n "id": 5,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "label_replace(\\n rate(ceph_rgw_op_get_obj_lat_sum{cluster=~\\"$cluster|\\", }[$__rate_interval]) /\\n rate(ceph_rgw_op_get_obj_lat_count{cluster=~\\"$cluster|\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", },\\n \\"rgw_host\\", \\"$1\\", \\"ceph_daemon\\", \\"rgw.(.*)\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{rgw_host}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "GET Latencies by RGW Instance",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "s",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "Total bytes transferred in/out of all radosgw instances within the cluster",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "bytes"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 6,\n "w": 8,\n "x": 0,\n "y": 8\n },\n "id": 6,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(rate(ceph_rgw_op_get_obj_bytes{cluster=~\\"$cluster|\\", }[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "GETs",\n "refId": "A"\n },\n {\n "expr": "sum(rate(ceph_rgw_op_put_obj_bytes{cluster=~\\"$cluster|\\", }[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "PUTs",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Bandwidth Consumed by Type",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "bytes",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "Total bytes transferred in/out through get/put operations, by radosgw instance",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "bytes"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 6,\n "w": 7,\n "x": 8,\n "y": 8\n },\n "id": 7,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "label_replace(sum by (instance_id) (\\n rate(ceph_rgw_op_get_obj_bytes{cluster=~\\"$cluster|\\", }[$__rate_interval]) +\\n rate(ceph_rgw_op_put_obj_bytes{cluster=~\\"$cluster|\\", }[$__rate_interval])) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", },\\n \\"rgw_host\\", \\"$1\\", \\"ceph_daemon\\", \\"rgw.(.*)\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{rgw_host}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Bandwidth by RGW Instance",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "bytes",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "Latencies are shown stacked, without a yaxis to provide a visual indication of PUT latency imbalance across RGW hosts",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "s"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 6,\n "w": 6,\n "x": 15,\n "y": 8\n },\n "id": 8,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "label_replace(\\n rate(ceph_rgw_op_put_obj_lat_sum{cluster=~\\"$cluster|\\", }[$__rate_interval]) /\\n rate(ceph_rgw_op_put_obj_lat_count{cluster=~\\"$cluster|\\", }[$__rate_interval]) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", },\\n \\"rgw_host\\", \\"$1\\", \\"ceph_daemon\\", \\"rgw.(.*)\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{rgw_host}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "PUT Latencies by RGW Instance",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "s",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 12,\n "w": 9,\n "x": 0,\n "y": 12\n },\n "id": 9,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "RGW Overview - HAProxy Metrics",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 12,\n "w": 5,\n "x": 0,\n "y": 12\n },\n "id": 10,\n "legend": {\n "alignAsTable": true,\n "avg": true,\n "current": true,\n "max": true,\n "min": true,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": true\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [\n [\n {\n "alias": "/.*Back.*/",\n "transform": "negative-Y"\n },\n {\n "alias": "/.*1.*/"\n },\n {\n "alias": "/.*2.*/"\n },\n {\n "alias": "/.*3.*/"\n },\n {\n "alias": "/.*4.*/"\n },\n {\n "alias": "/.*5.*/"\n },\n {\n "alias": "/.*other.*/"\n }\n ]\n ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(\\n rate(\\n haproxy_frontend_http_responses_total{code=~\\"$code\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\", proxy=~\\"frontend\\"}[$__rate_interval]\\n )\\n) by (code)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Frontend {{ code }}",\n "refId": "A"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_backend_http_responses_total{code=~\\"$code\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\", proxy=~\\"backend\\"}[$__rate_interval]\\n )\\n) by (code)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Backend {{ code }}",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Total responses by HTTP code",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 12,\n "w": 5,\n "x": 5,\n "y": 12\n },\n "id": 11,\n "legend": {\n "alignAsTable": true,\n "avg": true,\n "current": true,\n "max": true,\n "min": true,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": true\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [\n [\n {\n "alias": "/.*Response.*/",\n "transform": "negative-Y"\n },\n {\n "alias": "/.*Backend.*/",\n "transform": "negative-Y"\n }\n ]\n ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(\\n rate(\\n haproxy_frontend_http_requests_total{proxy=~\\"frontend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Requests",\n "refId": "A"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_backend_response_errors_total{proxy=~\\"backend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "Response errors",\n "refId": "B"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_frontend_request_errors_total{proxy=~\\"frontend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Requests errors",\n "refId": "C"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_backend_redispatch_warnings_total{proxy=~\\"backend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "Backend redispatch",\n "refId": "D"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_backend_retry_warnings_total{proxy=~\\"backend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "Backend retry",\n "refId": "E"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_frontend_requests_denied_total{proxy=~\\"frontend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "Request denied",\n "refId": "F"\n },\n {\n "expr": "sum(\\n haproxy_backend_current_queue{proxy=~\\"backend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "Backend Queued",\n "refId": "G"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Total requests / responses",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 12,\n "w": 5,\n "x": 10,\n "y": 12\n },\n "id": 12,\n "legend": {\n "alignAsTable": true,\n "avg": true,\n "current": true,\n "max": true,\n "min": true,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": true\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [\n [\n {\n "alias": "/.*Back.*/",\n "transform": "negative-Y"\n }\n ]\n ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(\\n rate(\\n haproxy_frontend_connections_total{proxy=~\\"frontend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Front",\n "refId": "A"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_backend_connection_attempts_total{proxy=~\\"backend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Back",\n "refId": "B"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_backend_connection_errors_total{proxy=~\\"backend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n )\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Back errors",\n "refId": "C"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Total number of connections",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 12,\n "w": 6,\n "x": 15,\n "y": 12\n },\n "id": 13,\n "legend": {\n "alignAsTable": true,\n "avg": true,\n "current": true,\n "max": true,\n "min": true,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": true\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [\n [\n {\n "alias": "/.*OUT.*/",\n "transform": "negative-Y"\n }\n ]\n ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(\\n rate(\\n haproxy_frontend_bytes_in_total{proxy=~\\"frontend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n ) * 8\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "IN Front",\n "refId": "A"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_frontend_bytes_out_total{proxy=~\\"frontend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n ) * 8\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "OUT Front",\n "refId": "B"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_backend_bytes_in_total{proxy=~\\"backend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n ) * 8\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "IN Back",\n "refId": "C"\n },\n {\n "expr": "sum(\\n rate(\\n haproxy_backend_bytes_out_total{proxy=~\\"backend\\", job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}[$__rate_interval]\\n ) * 8\\n) by (instance)\\n",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "OUT Back",\n "refId": "D"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Current total of incoming / outgoing bytes",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 27\n },\n "id": 14,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "RGW Overview - Bucket Notification",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "Shows the number of pending notifications in each Object topic queue, indicating how many messages are waiting to be delivered",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": ""\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 0,\n "y": 28\n },\n "id": 15,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "list",\n "placement": "right",\n "showLegend": true,\n "sortDesc": true\n }\n },\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "(\\n ceph_rgw_topic_persistent_topic_len\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{topic}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Pending Notifications by Topic",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "Shows the total size of pending notifications stored per Object topic, reflecting how much data is waiting to be delivered.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "bytes"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 12,\n "y": 28\n },\n "id": 16,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "list",\n "placement": "right",\n "showLegend": true,\n "sortDesc": true\n }\n },\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "(\\n ceph_rgw_topic_persistent_topic_size\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{topic}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Pending Notifications Size by Topic",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "bytes",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin",\n "overview"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": "",\n "multi": false,\n "name": "rgw_servers",\n "options": [ ],\n "query": "label_values(ceph_rgw_metadata{cluster=~\\"$cluster|\\", }, ceph_daemon)",\n "refresh": 1,\n "regex": ".*",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": "HTTP Code",\n "multi": false,\n "name": "code",\n "options": [ ],\n "query": "label_values(haproxy_server_http_responses_total{job=~\\"$job_haproxy\\", instance=~\\"$ingress_service\\"}, code)",\n "refresh": 1,\n "regex": "",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": ".+",\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": "job haproxy",\n "multi": true,\n "name": "job_haproxy",\n "options": [ ],\n "query": "label_values(haproxy_server_status, job)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": "Ingress Service",\n "multi": false,\n "name": "ingress_service",\n "options": [ ],\n "query": "label_values(haproxy_server_status{job=~\\"$job_haproxy\\"}, instance)",\n "refresh": 1,\n "regex": "",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "RGW Overview",\n "uid": "WAkugZpiz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/rbd-details.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.3.3"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "Detailed Performance of RBD Images (IOPS/Throughput/Latency)",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [\n {\n "asDropdown": true,\n "icon": "external link",\n "includeVars": true,\n "keepTime": true,\n "tags": [ ],\n "targetBlank": false,\n "title": "Browse Dashboards",\n "tooltip": "",\n "type": "dashboards",\n "url": ""\n }\n ],\n "panels": [\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "iops"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 9,\n "w": 8,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "rate(ceph_rbd_write_ops{pool=\\"$pool\\", image=\\"$image\\", cluster=~\\"$cluster|\\", }[$__rate_interval])",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{pool}} Write",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_rbd_read_ops{pool=\\"$pool\\", image=\\"$image\\", cluster=~\\"$cluster|\\", }[$__rate_interval])",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{pool}} Read",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "IOPS",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "iops",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "iops",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "Bps"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 9,\n "w": 8,\n "x": 8,\n "y": 0\n },\n "id": 3,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "rate(ceph_rbd_write_bytes{pool=\\"$pool\\", image=\\"$image\\", cluster=~\\"$cluster|\\", }[$__rate_interval])",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{pool}} Write",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_rbd_read_bytes{pool=\\"$pool\\", image=\\"$image\\", cluster=~\\"$cluster|\\", }[$__rate_interval])",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{pool}} Read",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Throughput",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "Bps",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "Bps",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "ns"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 9,\n "w": 8,\n "x": 16,\n "y": 0\n },\n "id": 4,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "rate(ceph_rbd_write_latency_sum{pool=\\"$pool\\", image=\\"$image\\", cluster=~\\"$cluster|\\", }[$__rate_interval]) /\\n rate(ceph_rbd_write_latency_count{pool=\\"$pool\\", image=\\"$image\\", cluster=~\\"$cluster|\\", }[$__rate_interval])\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{pool}} Write",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_rbd_read_latency_sum{pool=\\"$pool\\", image=\\"$image\\", cluster=~\\"$cluster|\\", }[$__rate_interval]) /\\n rate(ceph_rbd_read_latency_count{pool=\\"$pool\\", image=\\"$image\\", cluster=~\\"$cluster|\\", }[$__rate_interval])\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{pool}} Read",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Average Latency",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "ns",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "ns",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "",\n "multi": false,\n "name": "pool",\n "options": [ ],\n "query": "label_values(ceph_rbd_read_ops{cluster=~\\"$cluster|\\", }, pool)",\n "refresh": 1,\n "regex": "",\n "sort": 0,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "",\n "multi": false,\n "name": "image",\n "options": [ ],\n "query": "label_values(ceph_rbd_read_ops{cluster=~\\"$cluster|\\", pool=\\"$pool\\"}, image)",\n "refresh": 1,\n "regex": "",\n "sort": 0,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "RBD Details",\n "uid": "YhCYGcuZz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/ceph-application-overview.json': '{\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": {\n "type": "grafana",\n "uid": "-- Grafana --"\n },\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "target": {\n "limit": 100,\n "matchAny": false,\n "tags": [],\n "type": "dashboard"\n },\n "type": "dashboard"\n }\n ]\n },\n "editable": true,\n "fiscalYearStartMonth": 0,\n "graphTooltip": 0,\n "id": 14,\n "links": [],\n "panels": [\n {\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 11,\n "panels": [],\n "title": "Application (${application:csv})",\n "type": "row"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "mappings": [],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "bytes"\n },\n "overrides": []\n },\n "gridPos": {\n "h": 5,\n "w": 6,\n "x": 0,\n "y": 1\n },\n "id": 56,\n "options": {\n "colorMode": "value",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "auto",\n "percentChangeColorMode": "standard",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "showPercentChange": false,\n "textMode": "auto",\n "wideLayout": true\n },\n "pluginVersion": "11.6.0",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "expr": "sum(ceph_osd_stat_bytes{cluster=~\\"$cluster|\\"})",\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Cluster Capacity",\n "type": "stat"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "mappings": [],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "short"\n },\n "overrides": []\n },\n "gridPos": {\n "h": 5,\n "w": 6,\n "x": 6,\n "y": 1\n },\n "id": 52,\n "options": {\n "colorMode": "value",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "percentChangeColorMode": "standard",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "showPercentChange": false,\n "textMode": "auto",\n "wideLayout": true\n },\n "pluginVersion": "11.6.0",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "expr": "sum(rate(ceph_pool_wr{cluster=~\\"$cluster|\\",}[$__rate_interval]) *\\n on(pool_id) group_left(instance,name) ceph_pool_metadata{cluster=~\\"$cluster|\\",name=~\\"$pool\\"})",\n "legendFormat": "Write",\n "range": true,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "expr": "sum(rate(ceph_pool_rd{cluster=~\\"$cluster|\\",}[$__rate_interval]) *\\n on(pool_id) group_left(instance,name) ceph_pool_metadata{cluster=~\\"$cluster|\\",name=~\\"$pool\\"})",\n "hide": false,\n "legendFormat": "Read",\n "range": true,\n "refId": "B"\n }\n ],\n "title": "IOPS (${application:csv} - $pool)",\n "type": "stat"\n },\n {\n "fieldConfig": {\n "defaults": {},\n "overrides": []\n },\n "gridPos": {\n "h": 10,\n "w": 12,\n "x": 12,\n "y": 1\n },\n "id": 58,\n "options": {\n "alertInstanceLabelFilter": "{cluster=\\"$cluster\\"}",\n "alertName": "",\n "dashboardAlerts": false,\n "groupBy": [\n "cluster"\n ],\n "groupMode": "custom",\n "maxItems": 20,\n "showInactiveAlerts": false,\n "sortOrder": 1,\n "stateFilter": {\n "error": true,\n "firing": true,\n "noData": false,\n "normal": false,\n "pending": true\n },\n "viewMode": "list"\n },\n "pluginVersion": "11.6.0",\n "title": "Application Alerts",\n "type": "alertlist"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "mappings": [],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "bytes"\n },\n "overrides": []\n },\n "gridPos": {\n "h": 5,\n "w": 6,\n "x": 0,\n "y": 6\n },\n "id": 59,\n "options": {\n "displayMode": "gradient",\n "legend": {\n "calcs": [],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": false\n },\n "maxVizHeight": 300,\n "minVizHeight": 16,\n "minVizWidth": 8,\n "namePlacement": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "showUnfilled": true,\n "sizing": "auto",\n "valueMode": "color"\n },\n "pluginVersion": "11.6.0",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "exemplar": false,\n "expr": "sum by (application) (ceph_pool_stored{cluster=~\\"$cluster|\\",} *\\n on(pool_id) group_left(instance, name, application) ceph_pool_metadata{cluster=~\\"$cluster|\\",application=~\\"(.*Block.*)|(.*Filesystem.*)|(.*Object.*)\\"})",\n "format": "time_series",\n "instant": true,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Capacity Used",\n "type": "bargauge"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "mappings": [],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "short"\n },\n "overrides": []\n },\n "gridPos": {\n "h": 5,\n "w": 6,\n "x": 6,\n "y": 6\n },\n "id": 53,\n "options": {\n "colorMode": "value",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "percentChangeColorMode": "standard",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "showPercentChange": false,\n "textMode": "auto",\n "wideLayout": true\n },\n "pluginVersion": "11.6.0",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "expr": "sum(rate(ceph_pool_wr_bytes{cluster=~\\"$cluster|\\",}[$__rate_interval]) *\\n on(pool_id) group_left(instance,name) ceph_pool_metadata{cluster=~\\"$cluster|\\", application=~\\"$application\\", name=~\\"$pool\\"})",\n "legendFormat": "Write",\n "range": true,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "expr": "sum(rate(ceph_pool_rd_bytes{cluster=~\\"$cluster|\\",}[$__rate_interval]) *\\n on(pool_id) group_left(name) ceph_pool_metadata{cluster=~\\"$cluster|\\",application=~\\"$application\\", name=~\\"$pool\\"})",\n "hide": false,\n "legendFormat": "Read",\n "range": true,\n "refId": "B"\n }\n ],\n "title": "Throughput (${application:csv} - $pool)",\n "type": "stat"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "auto",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "mappings": [],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "name"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Pool Name"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #D"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "% Used"\n },\n {\n "id": "unit",\n "value": "percentunit"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #B"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Usable Free"\n },\n {\n "id": "unit",\n "value": "bytes"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #G"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "IOPS"\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #H"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Bandwidth"\n },\n {\n "id": "unit",\n "value": "Bps"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #J"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Stored"\n },\n {\n "id": "unit",\n "value": "bytes"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #M"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Quota"\n },\n {\n "id": "unit",\n "value": "bool_yes_no"\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #N"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Quota Bytes"\n },\n {\n "id": "unit",\n "value": "bytes"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 24,\n "x": 0,\n "y": 11\n },\n "id": 50,\n "options": {\n "cellHeight": "sm",\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true,\n "sortBy": [\n {\n "desc": true,\n "displayName": "% Used"\n }\n ]\n },\n "pluginVersion": "11.6.0",\n "targets": [\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "expr": "(ceph_pool_quota_bytes{cluster=~\\"$cluster|\\",}*\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{cluster=~\\"$cluster|\\",application=~\\"$application\\", name=~\\"$pool\\"}\\n - \\n (ceph_pool_stored{cluster=~\\"$cluster|\\",} *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{cluster=~\\"$cluster|\\",application=~\\"$application\\", name=~\\"$pool\\"})) > 0 \\n or\\n \\n(ceph_pool_max_avail{cluster=~\\"$cluster|\\",} *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{cluster=~\\"$cluster|\\",application=~\\"$application\\", name=~\\"$pool\\"})",\n "format": "table",\n "hide": false,\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "B",\n "refId": "B"\n },\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "expr": "(ceph_pool_stored{cluster=~\\"$cluster|\\",} *on(pool_id) group_left(instance, name) ceph_pool_metadata{cluster=~\\"$cluster|\\",name=~\\"$pool\\"})\\n /\\n (((ceph_pool_quota_bytes{cluster=~\\"$cluster|\\"} > 0) *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{cluster=~\\"$cluster|\\", name=~\\"$pool\\", application=~\\"$application\\"}) or \\n((ceph_pool_max_avail{cluster=~\\"$cluster|\\"} \\n+\\nceph_pool_stored{cluster=~\\"$cluster|\\", }) *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{cluster=~\\"$cluster|\\",name=~\\"$pool\\", application=~\\"$application\\"}))",\n "format": "table",\n "hide": false,\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "D",\n "refId": "D"\n },\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "expr": "rate(ceph_pool_rd{cluster=~\\"$cluster|\\", }[$__rate_interval]) *\\n on(pool_id) group_left(name) ceph_pool_metadata{cluster=~\\"$cluster|\\", application=~\\"$application\\", name=~\\"$pool\\"}\\n + rate(ceph_pool_wr{cluster=~\\"$cluster|\\", }[$__rate_interval]) *\\n on(pool_id) group_left(name) ceph_pool_metadata{cluster=~\\"$cluster|\\", application=~\\"$application\\", name=~\\"$pool\\"}\\n",\n "format": "table",\n "hide": false,\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "G",\n "refId": "G"\n },\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "expr": "rate(ceph_pool_rd_bytes{cluster=~\\"$cluster|\\", }[$__rate_interval]) *\\n on(pool_id) group_left(name) ceph_pool_metadata{cluster=~\\"$cluster|\\", application=~\\"$application\\", name=~\\"$pool\\"} +\\n rate(ceph_pool_wr_bytes{cluster=~\\"$cluster|\\", }[$__rate_interval]) *\\n on(pool_id) group_left(name) ceph_pool_metadata{cluster=~\\"$cluster|\\", application=~\\"$application\\", name=~\\"$pool\\"}\\n",\n "format": "table",\n "hide": false,\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "H",\n "refId": "H"\n },\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "expr": "ceph_pool_metadata{cluster=~\\"$cluster|\\", application=~\\"$application\\", name=~\\"$pool\\"}",\n "format": "table",\n "hide": false,\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "I",\n "refId": "I"\n },\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "expr": "ceph_pool_stored{cluster=~\\"$cluster|\\", } * on(pool_id) group_left ceph_pool_metadata{cluster=~\\"$cluster|\\", application=~\\"$application\\", name=~\\"$pool\\"}",\n "format": "table",\n "hide": false,\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "J",\n "refId": "J"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "exemplar": false,\n "expr": "ceph_pool_quota_bytes{cluster=~\\"$cluster|\\",} *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{cluster=~\\"$cluster|\\", application=~\\"$application\\",name=~\\"$pool\\"}",\n "format": "table",\n "hide": false,\n "instant": true,\n "legendFormat": "__auto",\n "range": false,\n "refId": "M"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "exemplar": false,\n "expr": "((ceph_pool_quota_bytes{cluster=~\\"$cluster|\\"}) *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{cluster=~\\"$cluster|\\", application=~\\"$application\\", name=~\\"$pool\\"})",\n "format": "table",\n "hide": false,\n "instant": true,\n "legendFormat": "N",\n "range": false,\n "refId": "N"\n }\n ],\n "title": "Pool Overview",\n "transformations": [\n {\n "id": "merge",\n "options": {}\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time": true,\n "Value #A": true,\n "Value #B": false,\n "Value #C": true,\n "Value #D": false,\n "Value #E": true,\n "Value #I": true,\n "Value #K": true,\n "__name__": true,\n "cluster": true,\n "compression_mode": true,\n "instance": true,\n "job": true,\n "pool_id": true,\n "type": true\n },\n "includeByName": {},\n "indexByName": {\n "Time": 0,\n "Value #A": 20,\n "Value #B": 10,\n "Value #C": 21,\n "Value #D": 12,\n "Value #E": 22,\n "Value #G": 13,\n "Value #H": 14,\n "Value #I": 18,\n "Value #J": 8,\n "Value #K": 19,\n "Value #M": 6,\n "Value #N": 11,\n "__name__": 15,\n "application": 4,\n "cluster": 1,\n "compression_mode": 16,\n "description": 5,\n "instance": 7,\n "job": 2,\n "name": 3,\n "pool_id": 9,\n "type": 17\n },\n "renameByName": {\n "Value #B": "",\n "Value #D": "",\n "Value #G": "",\n "Value #H": "",\n "Value #J": "",\n "Value #N": "",\n "__name__": "",\n "application": "Application",\n "compression_mode": "",\n "description": "Replication",\n "instance": "",\n "type": ""\n }\n }\n }\n ],\n "type": "table"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisBorderShow": false,\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "barWidthFactor": 0.6,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "insertNulls": false,\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "bytes"\n },\n "overrides": []\n },\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 0,\n "y": 19\n },\n "id": 23,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "right",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "hideZeros": false,\n "mode": "single",\n "sort": "none"\n }\n },\n "pluginVersion": "11.6.0",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "expr": "ceph_pool_stored{cluster=~\\"$cluster|\\",} *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{cluster=~\\"$cluster|\\",name=~\\"$pool\\"}",\n "legendFormat": "{{name}}",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Total used capacity",\n "type": "timeseries"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisBorderShow": false,\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "barWidthFactor": 0.6,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "insertNulls": false,\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "short"\n },\n "overrides": []\n },\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 12,\n "y": 19\n },\n "id": 17,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "right",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "hideZeros": false,\n "mode": "single",\n "sort": "none"\n }\n },\n "pluginVersion": "11.6.0",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "expr": "rate(ceph_pool_wr{cluster=~\\"$cluster|\\",}[$__rate_interval]) *\\n on(pool_id) group_left(instance,name) ceph_pool_metadata{cluster=~\\"$cluster|\\",name=~\\"$pool\\"}",\n "legendFormat": "{{name}} - Writes",\n "range": true,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "expr": "rate(ceph_pool_rd{cluster=~\\"$cluster|\\",}[$__rate_interval]) *\\n on(pool_id) group_left(instance,name) ceph_pool_metadata{cluster=~\\"$cluster|\\",name=~\\"$pool\\"}",\n "hide": false,\n "legendFormat": "{{name}} - Reads",\n "range": true,\n "refId": "B"\n }\n ],\n "title": "IOPS",\n "type": "timeseries"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisBorderShow": false,\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "barWidthFactor": 0.6,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "insertNulls": false,\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "bytes"\n },\n "overrides": []\n },\n "gridPos": {\n "h": 10,\n "w": 12,\n "x": 0,\n "y": 28\n },\n "id": 18,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "right",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "hideZeros": false,\n "mode": "single",\n "sort": "none"\n }\n },\n "pluginVersion": "11.6.0",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "expr": "rate(ceph_pool_wr_bytes{cluster=~\\"$cluster|\\",}[$__rate_interval]) *\\n on(pool_id) group_left(instance,name) ceph_pool_metadata{cluster=~\\"$cluster|\\", name=~\\"$pool\\"}",\n "legendFormat": "{{name}} - Writes",\n "range": true,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "expr": "rate(ceph_pool_rd_bytes{cluster=~\\"$cluster|\\",}[$__rate_interval]) *\\n on(pool_id) group_left(instance,name) ceph_pool_metadata{cluster=~\\"$cluster|\\", name=~\\"$pool\\"}",\n "hide": false,\n "legendFormat": "{{name}} - Reads",\n "range": true,\n "refId": "B"\n }\n ],\n "title": "Throughput",\n "type": "timeseries"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisBorderShow": false,\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "barWidthFactor": 0.6,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "insertNulls": false,\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "short"\n },\n "overrides": []\n },\n "gridPos": {\n "h": 10,\n "w": 12,\n "x": 12,\n "y": 28\n },\n "id": 24,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "right",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "hideZeros": false,\n "mode": "single",\n "sort": "none"\n }\n },\n "pluginVersion": "11.6.0",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "expr": "ceph_pool_objects{cluster=~\\"$cluster|\\",} *\\n on(pool_id) group_left(instance,name) ceph_pool_metadata{cluster=~\\"$cluster|\\", name=~\\"$pool\\"}\\n",\n "legendFormat": "{{name}}",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Total objects",\n "type": "timeseries"\n }\n ],\n "preload": false,\n "refresh": "30s",\n "schemaVersion": 41,\n "tags": [],\n "templating": {\n "list": [\n {\n "includeAll": false,\n "label": "Data Source",\n "name": "datasource",\n "options": [],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "definition": "label_values(ceph_health_status, cluster)",\n "includeAll": false,\n "label": "Cluster",\n "name": "cluster",\n "options": [],\n "query": {\n "query": "label_values(ceph_health_status, cluster)",\n "refId": "StandardVariableQuery"\n },\n "refresh": 2,\n "regex": "",\n "type": "query"\n },\n {\n "current": {\n "text": "All",\n "value": "$__all"\n },\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "definition": "label_values(ceph_pool_metadata{cluster=\\"$cluster\\"},application)",\n "includeAll": true,\n "label": "Application",\n "name": "application",\n "options": [],\n "query": {\n "qryType": 1,\n "query": "label_values(ceph_pool_metadata{cluster=\\"$cluster\\"},application)",\n "refId": "PrometheusVariableQueryEditor-VariableQuery"\n },\n "refresh": 2,\n "regex": "^(?=.*\\\\Filesystem\\\\b)|(?=.*\\\\Object\\\\b)|(?=.*\\\\Block\\\\b).*$",\n "sort": 6,\n "type": "query"\n },\n {\n "current": {\n "text": "All",\n "value": "$__all"\n },\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "definition": "label_values(ceph_pool_metadata{application=~\\"$application\\", cluster=\\"$cluster\\"},name)",\n "includeAll": true,\n "label": "Pool",\n "name": "pool",\n "options": [],\n "query": {\n "qryType": 1,\n "query": "label_values(ceph_pool_metadata{application=~\\"$application\\", cluster=\\"$cluster\\"},name)",\n "refId": "PrometheusVariableQueryEditor-VariableQuery"\n },\n "refresh": 2,\n "regex": "^[^.].*",\n "type": "query"\n }\n ]\n },\n "time": {\n "from": "now-6h",\n "to": "now"\n },\n "timepicker": {},\n "timezone": "",\n "title": "Ceph - Application Overview",\n "uid": "92LBIaJIz2",\n "version": 1\n}\n', '/etc/grafana/provisioning/dashboards/rgw-s3-analytics.json': '{\n "__inputs": [ ],\n "__requires": [ ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [\n {\n "asDropdown": true,\n "icon": "external link",\n "includeVars": true,\n "keepTime": true,\n "tags": [ ],\n "targetBlank": false,\n "title": "Browse Dashboards",\n "tooltip": "",\n "type": "dashboards",\n "url": ""\n }\n ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "Overview",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "colors": null,\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "decbytes"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 0,\n "y": 1\n },\n "id": 3,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_put_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Total PUTs",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "decbytes"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 6,\n "y": 1\n },\n "id": 4,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum\\n(ceph_rgw_op_get_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Total GETs",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "none"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 12,\n "y": 1\n },\n "id": 5,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_put_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Total Objects",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "decbytes"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 18,\n "y": 1\n },\n "id": 6,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum\\n((sum by(instance_id)(ceph_rgw_op_put_obj_bytes) > 0) / (sum by(instance_id)(ceph_rgw_op_put_obj_ops) > 0) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Average Object Size",\n "transparent": false,\n "type": "stat"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 0,\n "y": 4\n },\n "id": 7,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_list_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "List Objects",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_list_buckets_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "List Buckets",\n "range": true,\n "refId": "B"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_put_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Put Objects",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_per_bucket_get_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Get Objects",\n "range": false,\n "refId": "D"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_del_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Delete Objects",\n "range": false,\n "refId": "E"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_del_bucket_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Delete Buckets",\n "range": false,\n "refId": "F"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_copy_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Copy Objects",\n "range": true,\n "refId": "G"\n }\n ],\n "title": "Total Operations",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "decbytes"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 8,\n "y": 4\n },\n "id": 8,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_put_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Put Objects",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_per_bucket_get_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Get Objects",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_del_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Delete Objects",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_copy_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Copy Objects",\n "range": true,\n "refId": "D"\n }\n ],\n "title": "Total Size",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "ms"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 16,\n "y": 4\n },\n "id": 9,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_list_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "List Object",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_list_buckets_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "List Bucket",\n "range": true,\n "refId": "B"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_put_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Put Object",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_get_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Get Object",\n "range": false,\n "refId": "D"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_del_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Delete Object",\n "range": false,\n "refId": "E"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_del_bucket_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Delete Bucket",\n "range": false,\n "refId": "F"\n },\n {\n "datasource": "${datasource}",\n "expr": "sum(ceph_rgw_op_copy_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Copy Object",\n "range": true,\n "refId": "G"\n }\n ],\n "title": "Total Latencies",\n "type": "bargauge"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "auto",\n "cellOptions": {\n "type": "color-text"\n },\n "filterable": false,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "options": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "decbytes"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 0,\n "y": 12\n },\n "id": 10,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true,\n "sortBy": [\n {\n "desc": true,\n "displayName": "PUTs"\n }\n ]\n },\n "pluginVersion": "9.4.7",\n "styles": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_put_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Upload Objects",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_get_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Get Objects",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_del_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Delete Objects",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_copy_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Copy Objects",\n "range": false,\n "refId": "D"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Summary Per Bucket by Bandwidth",\n "transformations": [\n {\n "id": "merge",\n "options": { }\n },\n {\n "id": "groupBy",\n "options": {\n "fields": {\n "Bucket": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #A": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #B": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #D": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #F": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "bucket": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "ceph_daemon": {\n "aggregations": [ ],\n "operation": "groupby"\n }\n }\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time 1": true,\n "Time 2": true,\n "Time 3": true,\n "Time 4": true,\n "Time 5": true,\n "Time 6": true,\n "Time 7": true,\n "__name__ 1": true,\n "__name__ 2": true,\n "__name__ 3": true,\n "__name__ 4": true,\n "__name__ 5": true,\n "__name__ 6": true,\n "__name__ 7": true,\n "ceph_daemon 1": false,\n "ceph_daemon 2": true,\n "ceph_daemon 3": true,\n "ceph_daemon 4": true,\n "instance 1": true,\n "instance 2": true,\n "instance 3": true,\n "instance 4": true,\n "instance 5": true,\n "instance 6": true,\n "instance 7": true,\n "instance_id 1": true,\n "instance_id 2": true,\n "instance_id 3": true,\n "instance_id 4": true,\n "instance_id 5": true,\n "instance_id 6": true,\n "instance_id 7": true,\n "job 1": true,\n "job 2": true,\n "job 3": true,\n "job 4": true,\n "job 5": true,\n "job 6": true,\n "job 7": true\n },\n "indexByName": {\n "Value #A": 2,\n "Value #B": 3,\n "Value #D": 4,\n "Value #F": 5,\n "bucket": 1,\n "ceph_daemon": 0\n },\n "renameByName": {\n "Bucket": "",\n "Value #A": "PUTs",\n "Value #B": "GETs",\n "Value #C": "List",\n "Value #D": "Delete",\n "Value #E": "Copy",\n "Value #F": "Copy",\n "Value #G": "",\n "bucket": "Bucket",\n "ceph_daemon": "Daemon",\n "ceph_daemon 1": "Daemon"\n }\n }\n }\n ],\n "type": "table"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "auto",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": false,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "options": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "ms"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 12,\n "y": 12\n },\n "id": 11,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true,\n "sortBy": [\n {\n "desc": true,\n "displayName": "PUTs"\n }\n ]\n },\n "pluginVersion": "9.4.7",\n "styles": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_bucket_list_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "List Objects",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_bucket_put_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Upload Objects",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_bucket_get_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Get Objects",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_bucket_del_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Delete Objects",\n "range": false,\n "refId": "D"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_bucket_copy_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Copy Objects",\n "range": false,\n "refId": "E"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Latency(ms) Per Bucket",\n "transformations": [\n {\n "id": "merge",\n "options": { }\n },\n {\n "id": "joinByField",\n "options": {\n "byField": "Bucket",\n "mode": "outer"\n }\n },\n {\n "id": "groupBy",\n "options": {\n "fields": {\n "Bucket": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #A": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #B": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #C": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #D": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #F": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "bucket": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "ceph_daemon": {\n "aggregations": [ ],\n "operation": "groupby"\n }\n }\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time 1": true,\n "Time 2": true,\n "Time 3": true,\n "Time 4": true,\n "Time 5": true,\n "Time 6": true,\n "Time 7": true,\n "__name__ 1": true,\n "__name__ 2": true,\n "__name__ 3": true,\n "__name__ 4": true,\n "__name__ 5": true,\n "__name__ 6": true,\n "__name__ 7": true,\n "ceph_daemon 1": true,\n "ceph_daemon 2": true,\n "ceph_daemon 3": true,\n "ceph_daemon 4": true,\n "ceph_daemon 5": true,\n "instance 1": true,\n "instance 2": true,\n "instance 3": true,\n "instance 4": true,\n "instance 5": true,\n "instance 6": true,\n "instance 7": true,\n "instance_id 1": true,\n "instance_id 2": true,\n "instance_id 3": true,\n "instance_id 4": true,\n "instance_id 5": true,\n "instance_id 6": true,\n "instance_id 7": true,\n "job 1": true,\n "job 2": true,\n "job 3": true,\n "job 4": true,\n "job 5": true,\n "job 6": true,\n "job 7": true\n },\n "indexByName": {\n "Value #A": 2,\n "Value #B": 3,\n "Value #C": 4,\n "Value #D": 5,\n "Value #F": 6,\n "bucket": 1,\n "ceph_daemon": 0\n },\n "renameByName": {\n "Bucket": "",\n "Value #A": "PUTs",\n "Value #B": "GETs",\n "Value #C": "List",\n "Value #D": "Delete",\n "Value #E": "Copy",\n "Value #F": "Copy",\n "Value #G": "",\n "bucket": "Bucket",\n "ceph_daemon": "Daemon"\n }\n }\n }\n ],\n "type": "table"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "auto",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": false,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "options": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "decbytes"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 0,\n "y": 20\n },\n "id": 12,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true,\n "sortBy": [\n {\n "desc": true,\n "displayName": "PUTs"\n }\n ]\n },\n "pluginVersion": "9.4.7",\n "styles": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_put_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Upload Objects",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_get_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Get Objects",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_del_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Delete Objects",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_copy_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Copy Objects",\n "range": false,\n "refId": "D"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Summary Per User By Bandwidth",\n "transformations": [\n {\n "id": "merge",\n "options": { }\n },\n {\n "id": "groupBy",\n "options": {\n "fields": {\n "User": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #A": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #B": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #D": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #F": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "ceph_daemon": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "instance": {\n "aggregations": [ ]\n },\n "user": {\n "aggregations": [ ],\n "operation": "groupby"\n }\n }\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time 1": true,\n "Time 2": true,\n "Time 3": true,\n "Time 4": true,\n "Time 5": true,\n "Time 6": true,\n "Time 7": true,\n "__name__ 1": true,\n "__name__ 2": true,\n "__name__ 3": true,\n "__name__ 4": true,\n "__name__ 5": true,\n "__name__ 6": true,\n "__name__ 7": true,\n "ceph_daemon 1": true,\n "ceph_daemon 2": true,\n "ceph_daemon 3": true,\n "ceph_daemon 4": true,\n "instance 1": true,\n "instance 2": true,\n "instance 3": true,\n "instance 4": true,\n "instance 5": true,\n "instance 6": true,\n "instance 7": true,\n "instance_id 1": true,\n "instance_id 2": true,\n "instance_id 3": true,\n "instance_id 4": true,\n "instance_id 5": true,\n "instance_id 6": true,\n "instance_id 7": true,\n "job 1": true,\n "job 2": true,\n "job 3": true,\n "job 4": true,\n "job 5": true,\n "job 6": true,\n "job 7": true\n },\n "indexByName": {\n "Value #A": 2,\n "Value #B": 3,\n "Value #D": 4,\n "Value #F": 5,\n "ceph_daemon": 0,\n "user": 1\n },\n "renameByName": {\n "Bucket": "",\n "Value #A": "PUTs",\n "Value #B": "GETs",\n "Value #C": "List",\n "Value #D": "Delete",\n "Value #E": "Copy",\n "Value #F": "Copy",\n "Value #G": "",\n "ceph_daemon": "Daemon",\n "user": "User"\n }\n }\n }\n ],\n "type": "table"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "auto",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": false,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "options": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "ms"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 12,\n "y": 20\n },\n "id": 13,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true,\n "sortBy": [\n {\n "desc": true,\n "displayName": "PUTs"\n }\n ]\n },\n "pluginVersion": "9.4.7",\n "styles": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_list_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_put_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_get_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_del_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "D"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "ceph_rgw_op_per_user_copy_obj_lat_sum *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", }",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "E"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Latency(ms) Per User",\n "transformations": [\n {\n "id": "merge",\n "options": { }\n },\n {\n "id": "joinByField",\n "options": {\n "byField": "User",\n "mode": "outer"\n }\n },\n {\n "id": "groupBy",\n "options": {\n "fields": {\n "User": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #A": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #B": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #C": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #D": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #F": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "ceph_daemon": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "user": {\n "aggregations": [ ],\n "operation": "groupby"\n }\n }\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time 1": true,\n "Time 2": true,\n "Time 3": true,\n "Time 4": true,\n "Time 5": true,\n "Time 6": true,\n "Time 7": true,\n "__name__ 1": true,\n "__name__ 2": true,\n "__name__ 3": true,\n "__name__ 4": true,\n "__name__ 5": true,\n "__name__ 6": true,\n "__name__ 7": true,\n "ceph_daemon 1": true,\n "ceph_daemon 2": true,\n "ceph_daemon 3": true,\n "ceph_daemon 4": true,\n "ceph_daemon 5": true,\n "instance 1": true,\n "instance 2": true,\n "instance 3": true,\n "instance 4": true,\n "instance 5": true,\n "instance 6": true,\n "instance 7": true,\n "instance_id 1": true,\n "instance_id 2": true,\n "instance_id 3": true,\n "instance_id 4": true,\n "instance_id 5": true,\n "instance_id 6": true,\n "instance_id 7": true,\n "job 1": true,\n "job 2": true,\n "job 3": true,\n "job 4": true,\n "job 5": true,\n "job 6": true,\n "job 7": true\n },\n "indexByName": {\n "Value #A": 2,\n "Value #B": 3,\n "Value #C": 4,\n "Value #D": 5,\n "Value #F": 6,\n "ceph_daemon": 0,\n "user": 1\n },\n "renameByName": {\n "Bucket": "",\n "Value #A": "PUTs",\n "Value #B": "GETs",\n "Value #C": "List",\n "Value #D": "Delete",\n "Value #E": "Copy",\n "Value #F": "Copy",\n "Value #G": "",\n "ceph_daemon": "Daemon",\n "user": "User"\n }\n }\n }\n ],\n "type": "table"\n },\n {\n "collapse": false,\n "collapsed": true,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 28\n },\n "id": 14,\n "panels": [\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 0,\n "y": 29\n },\n "id": 15,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "topk(5, \\n sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_put_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })\\n)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Bucket PUTs by Operations",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 6,\n "y": 29\n },\n "id": 16,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "topk(5, \\n sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_get_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })\\n)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Bucket GETs by Operations",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 12,\n "y": 29\n },\n "id": 17,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [ ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "topk(5,\\n sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_put_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })\\n)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Buckets PUTs By Size",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 18,\n "y": 29\n },\n "id": 18,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [ ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "topk(5,\\n sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_get_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })\\n)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Buckets GETs By Size",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 0,\n "y": 37\n },\n "id": 19,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_put_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket PUTs by Size",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 6,\n "y": 37\n },\n "id": 20,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_get_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket GETs by Size",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 12,\n "y": 37\n },\n "id": 21,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_copy_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket Copy by Size",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 18,\n "y": 37\n },\n "id": 22,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_del_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket Delete by Size",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 0,\n "y": 45\n },\n "id": 23,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_get_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket GETs by Operations",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 6,\n "y": 45\n },\n "id": 24,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_put_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket PUTs by Operations",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 12,\n "y": 45\n },\n "id": 25,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_list_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket List by Operations",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 18,\n "y": 45\n },\n "id": 26,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_del_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket Delete by Operations",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 0,\n "y": 53\n },\n "id": 27,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (bucket, ceph_daemon) ((ceph_rgw_op_per_bucket_copy_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{bucket}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Bucket Copy by Operations",\n "type": "timeseries"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "auto",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": false,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "options": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "none"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 12,\n "y": 53\n },\n "id": 28,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true,\n "sortBy": [\n {\n "desc": true,\n "displayName": "PUTs"\n }\n ]\n },\n "pluginVersion": "9.4.7",\n "styles": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_put_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_get_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_del_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_copy_obj_bytes *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "D"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (bucket, ceph_daemon) (ceph_rgw_op_per_bucket_list_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "E"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Summary Per Bucket by Operations",\n "transformations": [\n {\n "id": "merge",\n "options": { }\n },\n {\n "id": "joinByField",\n "options": {\n "byField": "Bucket",\n "mode": "outer"\n }\n },\n {\n "id": "groupBy",\n "options": {\n "fields": {\n "Bucket": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #A": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #B": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #C": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #D": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #F": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "bucket": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "ceph_daemon": {\n "aggregations": [ ],\n "operation": "groupby"\n }\n }\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time 1": true,\n "Time 2": true,\n "Time 3": true,\n "Time 4": true,\n "Time 5": true,\n "Time 6": true,\n "Time 7": true,\n "__name__": true,\n "__name__ 1": true,\n "__name__ 2": true,\n "__name__ 3": true,\n "__name__ 4": true,\n "__name__ 5": true,\n "__name__ 6": true,\n "__name__ 7": true,\n "ceph_daemon 1": true,\n "ceph_daemon 2": true,\n "ceph_daemon 3": true,\n "ceph_daemon 4": true,\n "instance 1": true,\n "instance 2": true,\n "instance 3": true,\n "instance 4": true,\n "instance 5": true,\n "instance 6": true,\n "instance 7": true,\n "instance_id 1": true,\n "instance_id 2": true,\n "instance_id 3": true,\n "instance_id 4": true,\n "instance_id 5": true,\n "instance_id 6": true,\n "instance_id 7": true,\n "job 1": true,\n "job 2": true,\n "job 3": true,\n "job 4": true,\n "job 5": true,\n "job 6": true,\n "job 7": true\n },\n "indexByName": {\n "Value #A": 2,\n "Value #B": 3,\n "Value #C": 4,\n "Value #D": 5,\n "Value #F": 6,\n "bucket": 1,\n "ceph_daemon": 0\n },\n "renameByName": {\n "Bucket": "",\n "Value #A": "PUTs",\n "Value #B": "GETs",\n "Value #C": "List",\n "Value #D": "Delete",\n "Value #E": "Copy",\n "Value #F": "Copy",\n "Value #G": "",\n "bucket": "Bucket",\n "ceph_daemon": "Daemon"\n }\n }\n }\n ],\n "type": "table"\n }\n ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "Buckets",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "collapse": false,\n "collapsed": true,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 29\n },\n "id": 29,\n "panels": [\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 0,\n "y": 62\n },\n "id": 30,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "topk(5, \\n sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_put_obj_ops ) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })\\n)\\n",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Users PUTs By Operations",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 6,\n "y": 62\n },\n "id": 31,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "topk(5, \\n sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_get_obj_ops ) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })\\n)\\n",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Users GETs by Operations",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 12,\n "y": 62\n },\n "id": 32,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [ ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "topk(5, \\n sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_put_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })\\n)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Users PUTs by Size",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 18,\n "y": 62\n },\n "id": 33,\n "options": {\n "displayMode": "gradient",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [ ]\n }\n },\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "topk(5, \\n sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_get_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })\\n)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Users GETs By Size",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 0,\n "y": 70\n },\n "id": 34,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_put_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User PUTs by Size",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 6,\n "y": 70\n },\n "id": 35,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_get_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User GETs by Size",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 12,\n "y": 70\n },\n "id": 36,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_del_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User Delete by Size",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 18,\n "y": 70\n },\n "id": 37,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_copy_obj_bytes) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User COPY by Size",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 0,\n "y": 78\n },\n "id": 38,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_get_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User GETs by Operations",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 6,\n "y": 78\n },\n "id": 39,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_put_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User PUTs by Operations",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 12,\n "y": 78\n },\n "id": 40,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_list_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User List by Operations",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 6,\n "x": 18,\n "y": 78\n },\n "id": 41,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_del_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User Delete by Operations",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "color"\n },\n {\n "id": "color",\n "value": {\n "mode": "palette-classic"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 0,\n "y": 86\n },\n "id": 42,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "single",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "${datasource}",\n "expr": "sum by (user, ceph_daemon) ((ceph_rgw_op_per_user_copy_obj_ops) *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}} - {{user}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "User Copy by Operations",\n "type": "timeseries"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "auto",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": false,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "options": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "none"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 12,\n "y": 86\n },\n "id": 43,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true,\n "sortBy": [\n {\n "desc": true,\n "displayName": "PUTs"\n }\n ]\n },\n "pluginVersion": "9.4.7",\n "styles": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (user, ceph_daemon) (ceph_rgw_op_per_user_put_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (user, ceph_daemon) (ceph_rgw_op_per_user_get_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (user, ceph_daemon) (ceph_rgw_op_per_user_del_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (user, ceph_daemon) (ceph_rgw_op_per_user_copy_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "D"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "exemplar": false,\n "expr": "sum by (user, ceph_daemon) (ceph_rgw_op_per_user_list_obj_ops *\\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\\"$rgw_servers\\", cluster=~\\"$cluster|\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "E"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Summary Per User By Operations",\n "transformations": [\n {\n "id": "merge",\n "options": { }\n },\n {\n "id": "joinByField",\n "options": {\n "byField": "User",\n "mode": "outer"\n }\n },\n {\n "id": "groupBy",\n "options": {\n "fields": {\n "User": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #A": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #B": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #C": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #D": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "Value #F": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "ceph_daemon": {\n "aggregations": [ ],\n "operation": "groupby"\n },\n "user": {\n "aggregations": [ ],\n "operation": "groupby"\n }\n }\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": { },\n "indexByName": {\n "Value #A": 2,\n "Value #B": 3,\n "Value #C": 4,\n "Value #D": 5,\n "Value #F": 6,\n "ceph_daemon": 0,\n "user": 1\n },\n "renameByName": {\n "Value #A": "PUTs",\n "Value #B": "GETs",\n "Value #C": "LIST",\n "Value #D": "DELETE",\n "Value #F": "COPY",\n "ceph_daemon": "Daemon",\n "user": "User"\n }\n }\n }\n ],\n "type": "table"\n }\n ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "Users",\n "titleSize": "h6",\n "type": "row"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 22,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": null,\n "multi": false,\n "name": "rgw_servers",\n "options": [ ],\n "query": "label_values(ceph_rgw_metadata{cluster=~\\"$cluster|\\", }, ceph_daemon)",\n "refresh": 2,\n "regex": "",\n "sort": 0,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "datasource": "$datasource",\n "hide": 2,\n "label": "filters",\n "name": "Filters",\n "type": "adhoc"\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "RGW S3 Analytics",\n "uid": "BnxelG7Sz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/hosts-overview.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.3.2"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n },\n {\n "id": "singlestat",\n "name": "Singlestat",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [\n {\n "asDropdown": true,\n "icon": "external link",\n "includeVars": true,\n "keepTime": true,\n "tags": [ ],\n "targetBlank": false,\n "title": "Browse Dashboards",\n "tooltip": "",\n "type": "dashboards",\n "url": ""\n }\n ],\n "panels": [\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "format": "none",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 5,\n "w": 4,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "count(sum by (hostname) (ceph_osd_metadata{cluster=~\\"$cluster|\\", }))",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "OSD Hosts",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Average CPU busy across all hosts (OSD, RGW, MON etc) within the cluster",\n "format": "percentunit",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 5,\n "w": 4,\n "x": 4,\n "y": 0\n },\n "id": 3,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "avg(1 - (\\n avg by(instance) (\\n rate(node_cpu_seconds_total{mode=\'idle\',instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}[$__rate_interval]) or\\n rate(node_cpu{mode=\'idle\',instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}[$__rate_interval])\\n )\\n))\\n",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "AVG CPU Busy",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Average Memory Usage across all hosts in the cluster (excludes buffer/cache usage)",\n "format": "percentunit",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 5,\n "w": 4,\n "x": 8,\n "y": 0\n },\n "id": 4,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "avg ((\\n (\\n node_memory_MemTotal{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"} or\\n node_memory_MemTotal_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}\\n ) - ((\\n node_memory_MemFree{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"} or\\n node_memory_MemFree_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}) +\\n (\\n node_memory_Cached{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"} or\\n node_memory_Cached_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}\\n ) + (\\n node_memory_Buffers{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"} or\\n node_memory_Buffers_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}\\n ) + (\\n node_memory_Slab{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"} or\\n node_memory_Slab_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}\\n )\\n )\\n) / (\\n node_memory_MemTotal{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"} or\\n node_memory_MemTotal_bytes{instance=~\\"($osd_hosts|$rgw_hosts|$mon_hosts|$mds_hosts).*\\"}\\n))\\n",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "AVG RAM Utilization",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "IOPS Load at the device as reported by the OS on all OSD hosts",\n "format": "none",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 5,\n "w": 4,\n "x": 12,\n "y": 0\n },\n "id": 5,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum ((\\n rate(node_disk_reads_completed{instance=~\\"($osd_hosts).*\\"}[$__rate_interval]) or\\n rate(node_disk_reads_completed_total{instance=~\\"($osd_hosts).*\\"}[$__rate_interval])\\n) + (\\n rate(node_disk_writes_completed{instance=~\\"($osd_hosts).*\\"}[$__rate_interval]) or\\n rate(node_disk_writes_completed_total{instance=~\\"($osd_hosts).*\\"}[$__rate_interval])\\n))\\n",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Physical IOPS",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Average Disk utilization for all OSD data devices (i.e. excludes journal/WAL)",\n "format": "percent",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 5,\n "w": 4,\n "x": 16,\n "y": 0\n },\n "id": 6,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "avg (\\n label_replace(\\n (rate(node_disk_io_time_ms[$__rate_interval]) / 10 ) or\\n (rate(node_disk_io_time_seconds_total[$__rate_interval]) * 100),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^.:]*).*\\"\\n ) * on(instance, device) group_left(ceph_daemon) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{instance=~\\"($osd_hosts).*\\", cluster=~\\"$cluster|\\", },\\n \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^.:]*).*\\"\\n )\\n)\\n",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "AVG Disk Utilization",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Total send/receive network load across all hosts in the ceph cluster",\n "format": "bytes",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 5,\n "w": 4,\n "x": 20,\n "y": 0\n },\n "id": 7,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum (\\n (\\n rate(node_network_receive_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\",device!=\\"lo\\"}[$__rate_interval]) or\\n rate(node_network_receive_bytes_total{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\",device!=\\"lo\\"}[$__rate_interval])\\n ) unless on (device, instance)\\n label_replace((node_bonding_slaves > 0), \\"device\\", \\"$1\\", \\"master\\", \\"(.+)\\")\\n) +\\nsum (\\n (\\n rate(node_network_transmit_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\",device!=\\"lo\\"}[$__rate_interval]) or\\n rate(node_network_transmit_bytes_total{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\",device!=\\"lo\\"}[$__rate_interval])\\n ) unless on (device, instance)\\n label_replace((node_bonding_slaves > 0), \\"device\\", \\"$1\\", \\"master\\", \\"(.+)\\")\\n)\\n",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Network Load",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "percent"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 0,\n "y": 5\n },\n "id": 8,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "topk(10,\\n 100 * (\\n 1 - (\\n avg by(instance) (\\n rate(node_cpu_seconds_total{mode=\'idle\',instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}[$__rate_interval]) or\\n rate(node_cpu{mode=\'idle\',instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\"}[$__rate_interval])\\n )\\n )\\n )\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{instance}}",\n "refId": "A"\n }\n ],\n "title": "CPU Busy - Top 10 Hosts",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "Bps"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 12,\n "y": 5\n },\n "id": 9,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "topk(10, (sum by(instance) (\\n (\\n rate(node_network_receive_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\",device!=\\"lo\\"}[$__rate_interval]) or\\n rate(node_network_receive_bytes_total{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\",device!=\\"lo\\"}[$__rate_interval])\\n ) +\\n (\\n rate(node_network_transmit_bytes{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\",device!=\\"lo\\"}[$__rate_interval]) or\\n rate(node_network_transmit_bytes_total{instance=~\\"($osd_hosts|$mon_hosts|$mds_hosts|$rgw_hosts).*\\",device!=\\"lo\\"}[$__rate_interval])\\n ) unless on (device, instance)\\n label_replace((node_bonding_slaves > 0), \\"device\\", \\"$1\\", \\"master\\", \\"(.+)\\"))\\n))\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{instance}}",\n "refId": "A"\n }\n ],\n "title": "Network Load - Top 10 Hosts",\n "type": "timeseries"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": null,\n "multi": false,\n "name": "osd_hosts",\n "options": [ ],\n "query": "label_values(ceph_osd_metadata{cluster=~\\"$cluster|\\", }, hostname)",\n "refresh": 1,\n "regex": "([^.]*).*",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": null,\n "multi": false,\n "name": "mon_hosts",\n "options": [ ],\n "query": "label_values(ceph_mon_metadata{cluster=~\\"$cluster|\\", }, hostname)",\n "refresh": 1,\n "regex": "mon.(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": null,\n "multi": false,\n "name": "mds_hosts",\n "options": [ ],\n "query": "label_values(ceph_mds_inodes{hostname, cluster=~\\"$cluster|\\", })",\n "refresh": 1,\n "regex": "mds.(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": null,\n "multi": false,\n "name": "rgw_hosts",\n "options": [ ],\n "query": "label_values(ceph_rgw_metadata{hostname, cluster=~\\"$cluster|\\", })",\n "refresh": 1,\n "regex": "rgw.(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "Host Overview",\n "uid": "y0KGL0iZz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/pool-overview.json': '{\n "__inputs": [ ],\n "__requires": [ ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [\n {\n "asDropdown": true,\n "icon": "external link",\n "includeVars": true,\n "keepTime": true,\n "tags": [ ],\n "targetBlank": false,\n "title": "Browse Dashboards",\n "tooltip": "",\n "type": "dashboards",\n "url": ""\n }\n ],\n "panels": [\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "format": "none",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "count(ceph_pool_metadata{cluster=~\\"$cluster|\\", })",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Pools",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "avg"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Count of the pools that have compression enabled",\n "format": "none",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 3,\n "y": 0\n },\n "id": 3,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "count(ceph_pool_metadata{compression_mode!=\\"none\\", cluster=~\\"$cluster|\\", })",\n "format": "",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Pools with Compression",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Total raw capacity available to the cluster",\n "format": "bytes",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 6,\n "y": 0\n },\n "id": 4,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum(ceph_osd_stat_bytes{cluster=~\\"$cluster|\\", })",\n "format": "",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Total Raw Capacity",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Total raw capacity consumed by user data and associated overheads (metadata + redundancy)",\n "format": "bytes",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 9,\n "y": 0\n },\n "id": 5,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum(ceph_pool_bytes_used{cluster=~\\"$cluster|\\", })",\n "format": "",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Raw Capacity Consumed",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Total of client data stored in the cluster",\n "format": "bytes",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 12,\n "y": 0\n },\n "id": 6,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum(ceph_pool_stored{cluster=~\\"$cluster|\\", })",\n "format": "",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Logical Stored ",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "A compression saving is determined as the data eligible to be compressed minus the capacity used to store the data after compression",\n "format": "bytes",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 15,\n "y": 0\n },\n "id": 7,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum(\\n ceph_pool_compress_under_bytes{cluster=~\\"$cluster|\\", } -\\n ceph_pool_compress_bytes_used{cluster=~\\"$cluster|\\", }\\n)\\n",\n "format": "",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Compression Savings",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Indicates how suitable the data is within the pools that are/have been enabled for compression - averaged across all pools holding compressed data",\n "format": "percent",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 18,\n "y": 0\n },\n "id": 8,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "(\\n sum(ceph_pool_compress_under_bytes{cluster=~\\"$cluster|\\", } > 0) /\\n sum(ceph_pool_stored_raw{cluster=~\\"$cluster|\\", } and ceph_pool_compress_under_bytes{cluster=~\\"$cluster|\\", } > 0)\\n) * 100\\n",\n "format": "table",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Compression Eligibility",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "This factor describes the average ratio of data eligible to be compressed divided by the data actually stored. It does not account for data written that was ineligible for compression (too small, or compression yield too low)",\n "format": "none",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 21,\n "y": 0\n },\n "id": 9,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum(\\n ceph_pool_compress_under_bytes{cluster=~\\"$cluster|\\", } > 0)\\n / sum(ceph_pool_compress_bytes_used{cluster=~\\"$cluster|\\", } > 0\\n)\\n",\n "format": "",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "",\n "title": "Compression Factor",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "auto",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "Time"\n },\n "properties": [\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "instance"\n },\n "properties": [\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "job"\n },\n "properties": [\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "name"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Pool Name"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "pool_id"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Pool ID"\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #A"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Compression Factor"\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #D"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "% Used"\n },\n {\n "id": "unit",\n "value": "percentunit"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.cellOptions",\n "value": {\n "type": "color-text"\n }\n },\n {\n "id": "thresholds",\n "value": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(245, 54, 54, 0.9)",\n "value": null\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 70\n },\n {\n "color": "rgba(50, 172, 45, 0.97)",\n "value": 85\n }\n ]\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #B"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Usable Free"\n },\n {\n "id": "unit",\n "value": "bytes"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #C"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Compression Eligibility"\n },\n {\n "id": "unit",\n "value": "percent"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #E"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Compression Savings"\n },\n {\n "id": "unit",\n "value": "bytes"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #F"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Growth (5d)"\n },\n {\n "id": "unit",\n "value": "bytes"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.cellOptions",\n "value": {\n "type": "color-text"\n }\n },\n {\n "id": "thresholds",\n "value": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(245, 54, 54, 0.9)",\n "value": null\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 70\n },\n {\n "color": "rgba(50, 172, 45, 0.97)",\n "value": 85\n }\n ]\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #G"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "IOPS"\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #H"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Bandwidth"\n },\n {\n "id": "unit",\n "value": "Bps"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "__name__"\n },\n "properties": [\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "type"\n },\n "properties": [\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "compression_mode"\n },\n "properties": [\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "description"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Type"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #J"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Stored"\n },\n {\n "id": "unit",\n "value": "bytes"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #I"\n },\n "properties": [\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value #K"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Compression"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 6,\n "w": 24,\n "x": 0,\n "y": 3\n },\n "id": 10,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "10.4.0",\n "styles": [ ],\n "targets": [\n {\n "expr": "(\\n ceph_pool_compress_under_bytes{cluster=~\\"$cluster|\\", } /\\n ceph_pool_compress_bytes_used{cluster=~\\"$cluster|\\", } > 0\\n) and on(pool_id) (\\n (\\n (ceph_pool_compress_under_bytes{cluster=~\\"$cluster|\\", } > 0) /\\n ceph_pool_stored_raw{cluster=~\\"$cluster|\\", }\\n ) * 100 > 0.5\\n)\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "A",\n "refId": "A"\n },\n {\n "expr": "ceph_pool_max_avail{cluster=~\\"$cluster|\\", } *\\n on(pool_id) group_left(name) ceph_pool_metadata{cluster=~\\"$cluster|\\", }\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "B",\n "refId": "B"\n },\n {\n "expr": "(\\n (ceph_pool_compress_under_bytes{cluster=~\\"$cluster|\\", } > 0) /\\n ceph_pool_stored_raw{cluster=~\\"$cluster|\\", }\\n) * 100\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "C",\n "refId": "C"\n },\n {\n "expr": "ceph_pool_percent_used{cluster=~\\"$cluster|\\", } *\\n on(pool_id) group_left(name) ceph_pool_metadata{cluster=~\\"$cluster|\\", }\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "D",\n "refId": "D"\n },\n {\n "expr": "ceph_pool_compress_under_bytes{cluster=~\\"$cluster|\\", } -\\n ceph_pool_compress_bytes_used{cluster=~\\"$cluster|\\", } > 0\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "E",\n "refId": "E"\n },\n {\n "expr": "delta(ceph_pool_stored{cluster=~\\"$cluster|\\", }[5d])",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "F",\n "refId": "F"\n },\n {\n "expr": "rate(ceph_pool_rd{cluster=~\\"$cluster|\\", }[$__rate_interval])\\n + rate(ceph_pool_wr{cluster=~\\"$cluster|\\", }[$__rate_interval])\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "G",\n "refId": "G"\n },\n {\n "expr": "rate(ceph_pool_rd_bytes{cluster=~\\"$cluster|\\", }[$__rate_interval]) +\\n rate(ceph_pool_wr_bytes{cluster=~\\"$cluster|\\", }[$__rate_interval])\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "H",\n "refId": "H"\n },\n {\n "expr": "ceph_pool_metadata{cluster=~\\"$cluster|\\", }",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "I",\n "refId": "I"\n },\n {\n "expr": "ceph_pool_stored{cluster=~\\"$cluster|\\", } * on(pool_id) group_left ceph_pool_metadata{cluster=~\\"$cluster|\\", }",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "J",\n "refId": "J"\n },\n {\n "expr": "ceph_pool_metadata{compression_mode!=\\"none\\", cluster=~\\"$cluster|\\", }",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "K",\n "refId": "K"\n },\n {\n "expr": "",\n "format": "",\n "intervalFactor": "",\n "legendFormat": "L",\n "refId": "L"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Pool Overview",\n "transformations": [\n {\n "id": "merge",\n "options": { }\n },\n {\n "id": "seriesToRows",\n "options": { }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time": true,\n "Value #A": true,\n "Value #B": false,\n "Value #C": true,\n "Value #D": false,\n "Value #E": true,\n "Value #I": true,\n "Value #K": true,\n "__name__": true,\n "cluster": true,\n "compression_mode": true,\n "instance": true,\n "job": true,\n "pool_id": true,\n "type": true\n },\n "includeByName": { },\n "indexByName": { },\n "renameByName": { }\n }\n }\n ],\n "type": "table"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "IOPS",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 0,\n "y": 9\n },\n "id": 11,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "topk($topk,\\n round(\\n (\\n rate(ceph_pool_rd{cluster=~\\"$cluster|\\", }[$__rate_interval]) +\\n rate(ceph_pool_wr{cluster=~\\"$cluster|\\", }[$__rate_interval])\\n ), 1\\n ) * on(pool_id) group_left(instance,name) ceph_pool_metadata{cluster=~\\"$cluster|\\", }\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "refId": "A"\n },\n {\n "expr": "topk($topk,\\n rate(ceph_pool_wr{cluster=~\\"$cluster|\\", }[$__rate_interval]) +\\n on(pool_id) group_left(instance,name) ceph_pool_metadata{cluster=~\\"$cluster|\\", }\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{name}} - write",\n "refId": "B"\n }\n ],\n "title": "Top $topk Client IOPS by Pool",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Throughput",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "Bps"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 12,\n "y": 9\n },\n "id": 12,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "topk($topk,\\n (\\n rate(ceph_pool_rd_bytes{cluster=~\\"$cluster|\\", }[$__rate_interval]) +\\n rate(ceph_pool_wr_bytes{cluster=~\\"$cluster|\\", }[$__rate_interval])\\n ) * on(pool_id) group_left(instance, name) ceph_pool_metadata{cluster=~\\"$cluster|\\", }\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "refId": "A"\n }\n ],\n "title": "Top $topk Client Bandwidth by Pool",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Capacity Used",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "bytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 24,\n "x": 0,\n "y": 17\n },\n "id": 13,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "ceph_pool_bytes_used{cluster=~\\"$cluster|\\", } * on(pool_id) group_right ceph_pool_metadata{cluster=~\\"$cluster|\\", }",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "refId": "A"\n }\n ],\n "title": "Pool Capacity Usage (RAW)",\n "type": "timeseries"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 22,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": {\n "text": "15",\n "value": "15"\n },\n "hide": 0,\n "includeAll": false,\n "label": "TopK",\n "multi": false,\n "name": "topk",\n "options": [\n {\n "text": "15",\n "value": "15"\n }\n ],\n "query": "15",\n "refresh": 0,\n "type": "custom"\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "Ceph Pools Overview",\n "uid": "z99hzWtmk",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/radosgw-sync-overview.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.0.0"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [\n {\n "asDropdown": true,\n "icon": "external link",\n "includeVars": true,\n "keepTime": true,\n "tags": [ ],\n "targetBlank": false,\n "title": "Browse Dashboards",\n "tooltip": "",\n "type": "dashboards",\n "url": ""\n }\n ],\n "panels": [\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "Bps"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": true,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum by (source_zone) (rate(ceph_data_sync_from_zone_fetch_bytes_sum{cluster=~\\"$cluster|\\", }[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{source_zone}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Replication (throughput) from Source Zone",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "Bps",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 8,\n "y": 0\n },\n "id": 3,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": true,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum by (source_zone) (rate(ceph_data_sync_from_zone_fetch_bytes_count{cluster=~\\"$cluster|\\", }[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{source_zone}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Replication (objects) from Source Zone",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": "Objects/s",\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "ms"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 16,\n "y": 0\n },\n "id": 4,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": true,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum by (source_zone) (rate(ceph_data_sync_from_zone_poll_latency_sum{cluster=~\\"$cluster|\\", }[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{source_zone}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Polling Request Latency from Source Zone",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "ms",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 0,\n "y": 7\n },\n "id": 5,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": true,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum by (source_zone) (rate(ceph_data_sync_from_zone_fetch_errors{cluster=~\\"$cluster|\\", }[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{source_zone}}",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Unsuccessful Object Replications from Source Zone",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": "Count/s",\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 0,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "s"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 16,\n "x": 8,\n "y": 7\n },\n "id": 6,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "right",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "rate(ceph_rgw_sync_delta_sync_delta{instance_id=~\\"$rgw_servers\\", cluster=~\\"$cluster\\", }[$__rate_interval])",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{instance_id}} - {{shard_id}}",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Replication(Time) Delta per shard",\n "type": "timeseries"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin",\n "overview"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": null,\n "multi": false,\n "name": "rgw_servers",\n "options": [ ],\n "query": "label_values(ceph_rgw_metadata{cluster=~\\"$cluster|\\", }, ceph_daemon)",\n "refresh": 1,\n "regex": "rgw.(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "RGW Sync Overview",\n "uid": "rgw-sync-overview",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/cephfsdashboard.json': '{\n "__inputs": [ ],\n "__requires": [ ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": {\n "type": "datasource",\n "uid": "grafana"\n },\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "Ceph CephFS overview for official Ceph Prometheus plugin.",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [\n {\n "asDropdown": true,\n "icon": "external link",\n "includeVars": true,\n "keepTime": true,\n "tags": [ ],\n "targetBlank": false,\n "title": "Browse Dashboards",\n "tooltip": "",\n "type": "dashboards",\n "url": ""\n }\n ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "Summary",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "colors": null,\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": "null",\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "none"\n }\n },\n "gridPos": {\n "h": 8,\n "w": 3,\n "x": 0,\n "y": 1\n },\n "id": 3,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "count(ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"})",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A",\n "step": 60\n }\n ],\n "title": "Filesystems",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": "null",\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(50, 172, 45, 0.97)",\n "value": null\n }\n ]\n },\n "unit": "bytes"\n }\n },\n "gridPos": {\n "h": 4,\n "w": 3,\n "x": 3,\n "y": 1\n },\n "id": 4,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(ceph_pool_bytes_used{cluster=~\\"$cluster|\\", pool_id=~\\"$mdatapool\\"})",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A",\n "step": 60\n }\n ],\n "title": "Metadata used",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": "null",\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "Bps"\n }\n },\n "gridPos": {\n "h": 2,\n "w": 3,\n "x": 6,\n "y": 1\n },\n "id": 5,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(rate(ceph_pool_rd_bytes{cluster=~\\"$cluster|\\", pool_id=~\\"$mdatapool\\"}[$__rate_interval]))",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "Metadata bandwidth read",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": "null",\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(50, 172, 45, 0.97)",\n "value": null\n }\n ]\n },\n "unit": "bytes"\n }\n },\n "gridPos": {\n "h": 4,\n "w": 3,\n "x": 9,\n "y": 1\n },\n "id": 6,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(ceph_pool_bytes_used{cluster=~\\"$cluster|\\", pool_id=~\\"$datapool\\"})",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A",\n "step": 60\n }\n ],\n "title": "Data used",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": "null",\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "Bps"\n }\n },\n "gridPos": {\n "h": 2,\n "w": 3,\n "x": 12,\n "y": 1\n },\n "id": 7,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(rate(ceph_pool_rd_bytes{cluster=~\\"$cluster|\\", pool_id=~\\"$datapool\\"}[$__rate_interval]))",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "Data bandwidth read",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": "null",\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(50, 172, 45, 0.97)",\n "value": null\n }\n ]\n },\n "unit": "short"\n }\n },\n "gridPos": {\n "h": 4,\n "w": 3,\n "x": 15,\n "y": 1\n },\n "id": 8,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(rate(ceph_mds_server_handle_client_request{cluster=~\\"$cluster|\\", }[$__rate_interval]) * on(ceph_daemon) (ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"} * on(fs_id) group_left(ceph_daemon) ceph_mds_metadata{cluster=~\\"$cluster|\\", }))",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A",\n "step": 60\n }\n ],\n "title": "Client requests",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": "null",\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(50, 172, 45, 0.97)",\n "value": null\n }\n ]\n },\n "unit": "short"\n }\n },\n "gridPos": {\n "h": 4,\n "w": 3,\n "x": 18,\n "y": 1\n },\n "id": 9,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(ceph_mds_sessions_session_count{cluster=~\\"$cluster|\\", } * on(ceph_daemon) (ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"} * on(fs_id) group_left(ceph_daemon) ceph_mds_metadata{cluster=~\\"$cluster|\\", }))",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A",\n "step": 60\n }\n ],\n "title": "Client sessions",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": "null",\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(50, 172, 45, 0.97)",\n "value": null\n }\n ]\n },\n "unit": "short"\n }\n },\n "gridPos": {\n "h": 4,\n "w": 3,\n "x": 21,\n "y": 1\n },\n "id": 10,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(ceph_mds_inodes{cluster=~\\"$cluster|\\", } * on(ceph_daemon) (ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"} * on(fs_id) group_left(ceph_daemon) ceph_mds_metadata{cluster=~\\"$cluster|\\", }))",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A",\n "step": 60\n }\n ],\n "title": "MDS inodes",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": "null",\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "Bps"\n }\n },\n "gridPos": {\n "h": 2,\n "w": 3,\n "x": 6,\n "y": 3\n },\n "id": 11,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(rate(ceph_pool_wr_bytes{cluster=~\\"$cluster|\\", pool_id=~\\"$mdatapool\\"}[$__rate_interval]))",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "Metadata bandwidth write",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": "null",\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "Bps"\n }\n },\n "gridPos": {\n "h": 2,\n "w": 3,\n "x": 12,\n "y": 3\n },\n "id": 12,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(rate(ceph_pool_wr_bytes{cluster=~\\"$cluster|\\", pool_id=~\\"$datapool\\"}[$__rate_interval]))",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "Data bandwidth write",\n "transparent": false,\n "type": "stat"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": "null",\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "max": 100,\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(50, 172, 45, 0.97)",\n "value": null\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 70\n },\n {\n "color": "rgba(245, 54, 54, 0.9)",\n "value": 80\n }\n ]\n },\n "unit": "percent"\n }\n },\n "gridPos": {\n "h": 4,\n "w": 3,\n "x": 3,\n "y": 5\n },\n "id": 13,\n "interval": "1m",\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "showThresholdLabels": false,\n "showThresholdMarkers": true\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "(1 - (sum(ceph_pool_max_avail{cluster=~\\"$cluster|\\", pool_id=~\\"$mdatapool\\"}) / (sum(ceph_pool_bytes_used{cluster=~\\"$cluster|\\", pool_id=~\\"$mdatapool\\"}) + sum(ceph_pool_max_avail{cluster=~\\"$cluster|\\", pool_id=~\\"$mdatapool\\"})))) * 100",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "Metadata Used",\n "refId": "A",\n "step": 60\n }\n ],\n "title": "Metadata used (%)",\n "transparent": false,\n "type": "gauge"\n },\n {\n "colors": null,\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": "null",\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "#508642",\n "value": null\n }\n ]\n },\n "unit": "iops"\n }\n },\n "gridPos": {\n "h": 2,\n "w": 3,\n "x": 6,\n "y": 5\n },\n "id": 14,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(rate(ceph_pool_rd{cluster=~\\"$cluster|\\", pool_id=~\\"$mdatapool\\"}[$__rate_interval]))",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "Metadata IOPS read",\n "transparent": false,\n "type": "stat"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": "null",\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "max": 100,\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(50, 172, 45, 0.97)",\n "value": null\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 70\n },\n {\n "color": "rgba(245, 54, 54, 0.9)",\n "value": 80\n }\n ]\n },\n "unit": "percent"\n }\n },\n "gridPos": {\n "h": 4,\n "w": 3,\n "x": 9,\n "y": 5\n },\n "id": 15,\n "interval": "1m",\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "showThresholdLabels": false,\n "showThresholdMarkers": true\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "(1 - (sum(ceph_pool_max_avail{cluster=~\\"$cluster|\\", pool_id=~\\"$datapool\\"}) / (sum(ceph_pool_bytes_used{cluster=~\\"$cluster|\\", pool_id=~\\"$datapool\\"}) + sum(ceph_pool_max_avail{cluster=~\\"$cluster|\\", pool_id=~\\"$datapool\\"})))) * 100",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "Data Used",\n "refId": "A",\n "step": 60\n }\n ],\n "title": "Data used (%)",\n "transparent": false,\n "type": "gauge"\n },\n {\n "colors": null,\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": "null",\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "#508642",\n "value": null\n }\n ]\n },\n "unit": "iops"\n }\n },\n "gridPos": {\n "h": 2,\n "w": 3,\n "x": 12,\n "y": 5\n },\n "id": 16,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(rate(ceph_pool_rd{cluster=~\\"$cluster|\\", pool_id=~\\"$datapool\\"}[$__rate_interval]))",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "Data IOPS read",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": "null",\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(50, 172, 45, 0.97)",\n "value": null\n }\n ]\n },\n "unit": "short"\n }\n },\n "gridPos": {\n "h": 4,\n "w": 3,\n "x": 15,\n "y": 5\n },\n "id": 17,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(rate(ceph_mds_forward{cluster=~\\"$cluster|\\", }[$__rate_interval]) * on(ceph_daemon) (ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"} * on(fs_id) group_left(ceph_daemon) ceph_mds_metadata{cluster=~\\"$cluster|\\", }))",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A",\n "step": 60\n }\n ],\n "title": "Forward requests",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": "null",\n "result": {\n "index": 0,\n "text": "N/A"\n }\n },\n "type": "special"\n },\n {\n "id": 1,\n "options": {\n "match": "nan",\n "result": {\n "index": 1,\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(50, 172, 45, 0.97)",\n "value": null\n }\n ]\n },\n "unit": "ms"\n }\n },\n "gridPos": {\n "h": 4,\n "w": 3,\n "x": 18,\n "y": 5\n },\n "id": 18,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(increase(ceph_mds_reply_latency_sum{cluster=~\\"$cluster|\\", }[$__rate_interval]) * on(ceph_daemon) (ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"} * on(fs_id) group_left(ceph_daemon) ceph_mds_metadata{cluster=~\\"$cluster|\\", })) / sum(increase(ceph_mds_reply_latency_count{cluster=~\\"$cluster|\\", }[$__rate_interval]) * on(ceph_daemon) (ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"} * on(fs_id) group_left(ceph_daemon) ceph_mds_metadata{cluster=~\\"$cluster|\\", }))",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "range": true,\n "refId": "A",\n "step": 60\n }\n ],\n "title": "Reply latency",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": "null",\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(50, 172, 45, 0.97)",\n "value": null\n }\n ]\n },\n "unit": "short"\n }\n },\n "gridPos": {\n "h": 4,\n "w": 3,\n "x": 21,\n "y": 5\n },\n "id": 19,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(ceph_mds_caps{cluster=~\\"$cluster|\\", } * on(ceph_daemon) (ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"} * on(fs_id) group_left(ceph_daemon) ceph_mds_metadata{cluster=~\\"$cluster|\\", }))",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A",\n "step": 60\n }\n ],\n "title": "MDS caps",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": "null",\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "#508642",\n "value": null\n }\n ]\n },\n "unit": "iops"\n }\n },\n "gridPos": {\n "h": 2,\n "w": 3,\n "x": 6,\n "y": 7\n },\n "id": 20,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(rate(ceph_pool_wr{cluster=~\\"$cluster|\\", pool_id=~\\"$mdatapool\\"}[$__rate_interval]))",\n "format": "time_series",\n "instant": true,\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "Metadata IOPS write",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": "null",\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "#508642",\n "value": null\n }\n ]\n },\n "unit": "iops"\n }\n },\n "gridPos": {\n "h": 2,\n "w": 3,\n "x": 12,\n "y": 7\n },\n "id": 21,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(rate(ceph_pool_wr{cluster=~\\"$cluster|\\", pool_id=~\\"$datapool\\"}[$__rate_interval]))",\n "format": "time_series",\n "instant": true,\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "Data IOPS write",\n "transparent": false,\n "type": "stat"\n },\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 9\n },\n "id": 22,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "MDS",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [\n {\n "id": 0,\n "options": {\n "pattern": "(?<=\\\\.)(.*?)(?=\\\\.)",\n "result": {\n "index": 0,\n "text": "$1"\n }\n },\n "type": "regex"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 0,\n "y": 10\n },\n "id": 23,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.4.7",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "rate(ceph_mds_server_handle_client_request{cluster=~\\"$cluster|\\", }[$__rate_interval]) * on(ceph_daemon) (ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"} * on(fs_id) group_left(ceph_daemon) ceph_mds_metadata{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{ ceph_daemon }}",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Client requests",\n "type": "timeseries"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 8,\n "y": 10\n },\n "id": 24,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.4.7",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "rate(ceph_mds_forward{cluster=~\\"$cluster|\\", }[$__rate_interval]) * on(ceph_daemon) (ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"} * on(fs_id) group_left(ceph_daemon) ceph_mds_metadata{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{ ceph_daemon }}",\n "refId": "A"\n }\n ],\n "title": "Forward requests",\n "type": "timeseries"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 16,\n "y": 10\n },\n "id": 25,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.4.7",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "rate(ceph_mds_server_handle_slave_request{cluster=~\\"$cluster|\\", }[$__rate_interval]) * on(ceph_daemon) (ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"} * on(fs_id) group_left(ceph_daemon) ceph_mds_metadata{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{ ceph_daemon }}",\n "refId": "A"\n }\n ],\n "title": "Slave requests",\n "type": "timeseries"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 0,\n "y": 17\n },\n "id": 26,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.4.7",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "ceph_mds_sessions_session_count{cluster=~\\"$cluster|\\", } * on(ceph_daemon) (ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"} * on(fs_id) group_left(ceph_daemon) ceph_mds_metadata{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{ ceph_daemon }}",\n "refId": "A"\n }\n ],\n "title": "Session count",\n "type": "timeseries"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "ms"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 8,\n "y": 17\n },\n "id": 27,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.4.7",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum by (ceph_daemon) (increase(ceph_mds_reply_latency_sum{cluster=~\\"$cluster|\\", }[$__rate_interval]) * on(ceph_daemon) (ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"} * on(fs_id) group_left(ceph_daemon) ceph_mds_metadata{cluster=~\\"$cluster|\\", })) / sum by (ceph_daemon) (increase(ceph_mds_reply_latency_count{cluster=~\\"$cluster|\\", }[$__rate_interval]) * on(ceph_daemon) (ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"} * on(fs_id) group_left(ceph_daemon) ceph_mds_metadata{cluster=~\\"$cluster|\\", }))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{ ceph_daemon }}",\n "refId": "A"\n }\n ],\n "title": "Reply latency",\n "type": "timeseries"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Reads(-) / Writes (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [ ],\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "none"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 16,\n "y": 17\n },\n "id": 28,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.4.7",\n "seriesOverrides": [\n {\n "alias": "/.*Reads/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "sum(rate(ceph_objecter_op_r{cluster=~\\"$cluster|\\", }[$__rate_interval]) * on(ceph_daemon) (ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"} * on(fs_id) group_left(ceph_daemon) ceph_mds_metadata{cluster=~\\"$cluster|\\", }))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Read Ops",\n "refId": "A"\n },\n {\n "expr": "sum(rate(ceph_objecter_op_w{cluster=~\\"$cluster|\\", }[$__rate_interval]) * on(ceph_daemon) (ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"} * on(fs_id) group_left(ceph_daemon) ceph_mds_metadata{cluster=~\\"$cluster|\\", }))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Write Ops",\n "refId": "B"\n }\n ],\n "title": "Workload",\n "type": "timeseries"\n },\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 24\n },\n "id": 29,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "Log",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 0,\n "y": 25\n },\n "id": 30,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.4.7",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "rate(ceph_mds_log_evadd{cluster=~\\"$cluster|\\", }[$__rate_interval]) * on(ceph_daemon) (ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"} * on(fs_id) group_left(ceph_daemon) ceph_mds_metadata{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{ ceph_daemon }}",\n "refId": "A"\n }\n ],\n "title": "Log submit",\n "type": "timeseries"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 8,\n "y": 25\n },\n "id": 31,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.4.7",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "ceph_mds_log_ev{cluster=~\\"$cluster|\\", } * on(ceph_daemon) (ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"} * on(fs_id) group_left(ceph_daemon) ceph_mds_metadata{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{ ceph_daemon }}",\n "refId": "A"\n }\n ],\n "title": "Log events",\n "type": "timeseries"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 16,\n "y": 25\n },\n "id": 32,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.4.7",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "ceph_mds_log_seg{cluster=~\\"$cluster|\\", } * on(ceph_daemon) (ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"} * on(fs_id) group_left(ceph_daemon) ceph_mds_metadata{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{ ceph_daemon }}",\n "refId": "A"\n }\n ],\n "title": "Log segments",\n "type": "timeseries"\n },\n {\n "collapse": false,\n "collapsed": true,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 32\n },\n "id": 33,\n "panels": [\n {\n "datasource": {\n "type": "datasource",\n "uid": "grafana"\n },\n "gridPos": {\n "h": 8,\n "w": 24,\n "x": 0,\n "y": 33\n },\n "id": 34,\n "limit": 10,\n "onlyAlertsOnDashboard": true,\n "options": {\n "alertInstanceLabelFilter": "",\n "alertName": "CephFilesystem",\n "dashboardAlerts": false,\n "groupBy": [ ],\n "groupMode": "default",\n "maxItems": 20,\n "sortOrder": 3,\n "stateFilter": {\n "error": true,\n "firing": true,\n "noData": false,\n "normal": false,\n "pending": true\n },\n "viewMode": "list"\n },\n "show": "current",\n "sortOrder": 1,\n "stateFilter": [ ],\n "title": "Active alerts",\n "type": "alertlist"\n }\n ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "Alerts",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "collapse": false,\n "collapsed": true,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 41\n },\n "id": 35,\n "panels": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 0,\n "y": 42\n },\n "id": 36,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.4.7",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "ceph_mds_mem_ino{cluster=~\\"$cluster|\\", } * on(ceph_daemon) (ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"} * on(fs_id) group_left(ceph_daemon) ceph_mds_metadata{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{ ceph_daemon }}",\n "refId": "A"\n }\n ],\n "title": "Inodes",\n "type": "timeseries"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 8,\n "y": 42\n },\n "id": 37,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.4.7",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "ceph_mds_exported_inodes{cluster=~\\"$cluster|\\", } * on(ceph_daemon) (ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"} * on(fs_id) group_left(ceph_daemon) ceph_mds_metadata{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{ ceph_daemon }}",\n "refId": "A"\n }\n ],\n "title": "Exported inodes",\n "type": "timeseries"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 16,\n "y": 42\n },\n "id": 38,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.4.7",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "ceph_mds_imported_inodes{cluster=~\\"$cluster|\\", } * on(ceph_daemon) (ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"} * on(fs_id) group_left(ceph_daemon) ceph_mds_metadata{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{ ceph_daemon }}",\n "refId": "A"\n }\n ],\n "title": "Imported inodes",\n "type": "timeseries"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 0,\n "y": 49\n },\n "id": 39,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.4.7",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "ceph_mds_mem_dn{cluster=~\\"$cluster|\\", } * on(ceph_daemon) (ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"} * on(fs_id) group_left(ceph_daemon) ceph_mds_metadata{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{ ceph_daemon }}",\n "refId": "A"\n }\n ],\n "title": "Dentries",\n "type": "timeseries"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 8,\n "y": 49\n },\n "id": 40,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.4.7",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "ceph_mds_caps{cluster=~\\"$cluster|\\", } * on(ceph_daemon) (ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"} * on(fs_id) group_left(ceph_daemon) ceph_mds_metadata{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{ ceph_daemon }}",\n "refId": "A"\n }\n ],\n "title": "Caps",\n "type": "timeseries"\n }\n ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "Memory",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "collapse": false,\n "collapsed": true,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 56\n },\n "id": 41,\n "panels": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 40,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 0,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [ ],\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "bytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byRegexp",\n "options": "/^Total.*$/"\n },\n "properties": [\n {\n "id": "custom.fillOpacity",\n "value": 0\n },\n {\n "id": "custom.lineWidth",\n "value": 4\n },\n {\n "id": "custom.stacking",\n "value": {\n "group": false,\n "mode": "normal"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byRegexp",\n "options": "/^Raw.*$/"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "#BF1B00",\n "mode": "fixed"\n }\n },\n {\n "id": "custom.fillOpacity",\n "value": 0\n },\n {\n "id": "custom.lineWidth",\n "value": 4\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 7,\n "w": 14,\n "x": 0,\n "y": 57\n },\n "id": 42,\n "options": {\n "legend": {\n "calcs": [\n "mean",\n "lastNotNull",\n "max"\n ],\n "displayMode": "table",\n "placement": "right",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.4.7",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(ceph_pool_max_avail{cluster=~\\"$cluster|\\", pool_id=~\\"^$mdatapool$\\"})",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "Available",\n "metric": "ceph_pool_available_bytes",\n "range": true,\n "refId": "A",\n "step": 60\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(ceph_pool_bytes_used{cluster=~\\"$cluster|\\", pool_id=~\\"^$mdatapool$\\"})",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "Used",\n "metric": "ceph_pool",\n "refId": "B",\n "step": 60\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(ceph_pool_bytes_used{cluster=~\\"$cluster|\\", pool_id=~\\"^$mdatapool$\\"}) + sum(ceph_pool_max_avail{cluster=~\\"$cluster|\\", pool_id=~\\"^$mdatapool$\\"})",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "Total",\n "metric": "ceph_pool",\n "refId": "C",\n "step": 60\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(ceph_pool_raw_bytes_used{cluster=~\\"$cluster|\\", pool_id=~\\"^$mdatapool$\\"})",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "Raw Used",\n "metric": "ceph_pool",\n "refId": "D",\n "step": 60\n }\n ],\n "title": "Pool storage",\n "type": "timeseries"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 2,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [ ],\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 6,\n "x": 14,\n "y": 57\n },\n "id": 43,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.4.7",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(ceph_pool_objects{cluster=~\\"$cluster|\\", pool_id=~\\"$mdatapool\\"})",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "Objects",\n "refId": "A",\n "step": 60\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(ceph_pool_dirty{cluster=~\\"$cluster|\\", pool_id=~\\"$mdatapool\\"})",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "Dirty Objects",\n "refId": "B",\n "step": 60\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(ceph_pool_quota_objects{cluster=~\\"$cluster|\\", pool_id=~\\"$mdatapool\\"})",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "Quota Objects",\n "refId": "C"\n }\n ],\n "title": "Objects in pool",\n "type": "timeseries"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": "null",\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "max": 1,\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "percentunit"\n }\n },\n "gridPos": {\n "h": 7,\n "w": 4,\n "x": 20,\n "y": 57\n },\n "id": 44,\n "interval": "1m",\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "showThresholdLabels": false,\n "showThresholdMarkers": true\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(ceph_pool_bytes_used{cluster=~\\"$cluster|\\", pool_id=~\\"$mdatapool\\"}) / sum(ceph_pool_bytes_used{cluster=~\\"$cluster|\\", pool_id=~\\"$mdatapool\\"} + ceph_pool_max_avail{cluster=~\\"$cluster|\\", pool_id=~\\"$mdatapool\\"})",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A",\n "step": 60\n }\n ],\n "title": "Usage",\n "transparent": false,\n "type": "gauge"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 2,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [ ],\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 12,\n "x": 0,\n "y": 64\n },\n "id": 45,\n "options": {\n "legend": {\n "calcs": [\n "mean",\n "lastNotNull",\n "max",\n "min"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.4.7",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(rate(ceph_pool_rd{cluster=~\\"$cluster|\\", pool_id=~\\"$mdatapool\\"}[$__rate_interval]))",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "Read",\n "refId": "A",\n "step": 60\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(rate(ceph_pool_wr{cluster=~\\"$cluster|\\", pool_id=~\\"$mdatapool\\"}[$__rate_interval]))",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "Write",\n "refId": "B",\n "step": 60\n }\n ],\n "title": "IOPS",\n "type": "timeseries"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 2,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [ ],\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 12,\n "x": 12,\n "y": 64\n },\n "id": 46,\n "options": {\n "legend": {\n "calcs": [\n "mean",\n "lastNotNull",\n "max",\n "min"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.4.7",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(rate(ceph_pool_rd_bytes{cluster=~\\"$cluster|\\", pool_id=~\\"$mdatapool\\"}[$__rate_interval]))",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "Read Bytes",\n "refId": "A",\n "step": 60\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(rate(ceph_pool_wr_bytes{cluster=~\\"$cluster|\\", pool_id=~\\"$mdatapool\\"}[$__rate_interval]))",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "Write Bytes",\n "refId": "B",\n "step": 60\n }\n ],\n "title": "Throughput",\n "type": "timeseries"\n }\n ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "Metadata Pool",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "collapse": false,\n "collapsed": true,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 71\n },\n "id": 47,\n "panels": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 40,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [ ],\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "bytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byRegexp",\n "options": "/^Total.*$/"\n },\n "properties": [\n {\n "id": "custom.fillOpacity",\n "value": 0\n },\n {\n "id": "custom.lineWidth",\n "value": 4\n },\n {\n "id": "custom.stacking",\n "value": {\n "group": false,\n "mode": "normal"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byRegexp",\n "options": "/^Raw.*$/"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "#BF1B00",\n "mode": "fixed"\n }\n },\n {\n "id": "custom.fillOpacity",\n "value": 0\n },\n {\n "id": "custom.lineWidth",\n "value": 4\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 7,\n "w": 14,\n "x": 0,\n "y": 72\n },\n "id": 48,\n "options": {\n "legend": {\n "calcs": [\n "mean",\n "lastNotNull",\n "max"\n ],\n "displayMode": "table",\n "placement": "right",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.4.7",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(ceph_pool_max_avail{cluster=~\\"$cluster|\\", pool_id=~\\"^$datapool$\\"})",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "Available",\n "metric": "ceph_pool_available_bytes",\n "range": true,\n "refId": "A",\n "step": 60\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(ceph_pool_bytes_used{cluster=~\\"$cluster|\\", pool_id=~\\"^$datapool$\\"})",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "Used",\n "metric": "ceph_pool",\n "refId": "B",\n "step": 60\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(ceph_pool_bytes_used{cluster=~\\"$cluster|\\", pool_id=~\\"^$datapool$\\"}) + sum(ceph_pool_max_avail{cluster=~\\"$cluster|\\", pool_id=~\\"^$mdatapool$\\"})",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "Total",\n "metric": "ceph_pool",\n "refId": "C",\n "step": 60\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(ceph_pool_raw_bytes_used{cluster=~\\"$cluster|\\", pool_id=~\\"^$datapool$\\"})",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "Raw Used",\n "metric": "ceph_pool",\n "refId": "D",\n "step": 60\n }\n ],\n "title": "Pool storage",\n "type": "timeseries"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 2,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [ ],\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 6,\n "x": 14,\n "y": 72\n },\n "id": 49,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.4.7",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(ceph_pool_objects{cluster=~\\"$cluster|\\", pool_id=~\\"$datapool\\"})",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "Objects",\n "refId": "A",\n "step": 60\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(ceph_pool_dirty{cluster=~\\"$cluster|\\", pool_id=~\\"$datapool\\"})",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "Dirty Objects",\n "refId": "B",\n "step": 60\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(ceph_pool_quota_objects{cluster=~\\"$cluster|\\", pool_id=~\\"$datapool\\"})",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "Quota Objects",\n "refId": "C"\n }\n ],\n "title": "Objects in pool",\n "type": "timeseries"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": "null",\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "max": 1,\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "percentunit"\n }\n },\n "gridPos": {\n "h": 7,\n "w": 4,\n "x": 20,\n "y": 72\n },\n "id": 50,\n "interval": "1m",\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "showThresholdLabels": false,\n "showThresholdMarkers": true\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(ceph_pool_bytes_used{cluster=~\\"$cluster|\\", pool_id=~\\"$datapool\\"}) / sum(ceph_pool_bytes_used{cluster=~\\"$cluster|\\", pool_id=~\\"$datapool\\"} + ceph_pool_max_avail{cluster=~\\"$cluster|\\", pool_id=~\\"$datapool\\"})",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A",\n "step": 60\n }\n ],\n "title": "Usage",\n "transparent": false,\n "type": "gauge"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 2,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [ ],\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 12,\n "x": 0,\n "y": 79\n },\n "id": 51,\n "options": {\n "legend": {\n "calcs": [\n "mean",\n "lastNotNull",\n "max",\n "min"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.4.7",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(rate(ceph_pool_rd{cluster=~\\"$cluster|\\", pool_id=~\\"$datapool\\"}[$__rate_interval]))",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "Read",\n "refId": "A",\n "step": 60\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(rate(ceph_pool_wr{cluster=~\\"$cluster|\\", pool_id=~\\"$datapool\\"}[$__rate_interval]))",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "Write",\n "refId": "B",\n "step": 60\n }\n ],\n "title": "IOPS",\n "type": "timeseries"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 2,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "mappings": [ ],\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 12,\n "x": 12,\n "y": 79\n },\n "id": 52,\n "options": {\n "legend": {\n "calcs": [\n "mean",\n "lastNotNull",\n "max",\n "min"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.4.7",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(rate(ceph_pool_rd_bytes{cluster=~\\"$cluster|\\", pool_id=~\\"$datapool\\"}[$__rate_interval]))",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "Read Bytes",\n "refId": "A",\n "step": 60\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "expr": "sum(rate(ceph_pool_wr_bytes{cluster=~\\"$cluster|\\", pool_id=~\\"$datapool\\"}[$__rate_interval]))",\n "format": "time_series",\n "interval": "$__rate_interval",\n "intervalFactor": 1,\n "legendFormat": "Write Bytes",\n "refId": "B",\n "step": 60\n }\n ],\n "title": "Throughput",\n "type": "timeseries"\n }\n ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "Data Pool",\n "titleSize": "h6",\n "type": "row"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 38,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": {\n "text": {\n "selected": false,\n "text": "a",\n "value": "a"\n },\n "value": {\n "selected": false,\n "text": "a",\n "value": "a"\n }\n },\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "hide": 0,\n "includeAll": true,\n "label": "Filesystem",\n "multi": false,\n "name": "name",\n "options": [ ],\n "query": {\n "query": "label_values(ceph_fs_metadata{cluster=~\\"$cluster|\\", }, name)",\n "refId": "StandardVariableQuery"\n },\n "refresh": 1,\n "regex": "",\n "sort": 0,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": {\n "text": {\n "selected": false,\n "text": "All",\n "value": "$__all"\n },\n "value": {\n "selected": false,\n "text": "All",\n "value": "$__all"\n }\n },\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "hide": 2,\n "includeAll": true,\n "label": "metadata pool",\n "multi": false,\n "name": "mdatapool",\n "options": [ ],\n "query": {\n "query": "label_values(ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"}, metadata_pool)",\n "refId": "StandardVariableQuery"\n },\n "refresh": 1,\n "regex": "",\n "sort": 0,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": {\n "text": {\n "selected": false,\n "text": "All",\n "value": "$__all"\n },\n "value": {\n "selected": false,\n "text": "All",\n "value": "$__all"\n }\n },\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "hide": 2,\n "includeAll": true,\n "label": "data pool",\n "multi": false,\n "name": "datapool",\n "options": [ ],\n "query": {\n "query": "label_values(ceph_fs_metadata{cluster=~\\"$cluster|\\", name=~\\"$name\\"}, data_pools)",\n "refId": "StandardVariableQuery"\n },\n "refresh": 1,\n "regex": "",\n "sort": 0,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-6h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "Ceph - Filesystem Overview",\n "uid": "718Bruins",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/ceph-cluster-advanced.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.3.2"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n },\n {\n "id": "heatmap",\n "name": "Heatmap",\n "type": "panel",\n "version": "5.0.0"\n },\n {\n "id": "singlestat",\n "name": "Singlestat",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "Ceph cluster overview",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [\n {\n "asDropdown": true,\n "icon": "external link",\n "includeVars": true,\n "keepTime": true,\n "tags": [ ],\n "targetBlank": false,\n "title": "Browse Dashboards",\n "tooltip": "",\n "type": "dashboards",\n "url": ""\n }\n ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "CLUSTER STATE",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "0": {\n "text": "HEALTHY"\n },\n "1": {\n "text": "WARNING"\n },\n "2": {\n "text": "ERROR"\n }\n },\n "type": "value"\n },\n {\n "id": 1,\n "options": {\n "match": null,\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "#9ac48a"\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 1\n },\n {\n "color": "rgba(245, 54, 54, 0.9)",\n "value": 2\n }\n ]\n },\n "unit": "none"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 0,\n "y": 1\n },\n "id": 3,\n "interval": "1m",\n "links": [ ],\n "options": {\n "colorMode": "value",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "ceph_health_status{cluster=~\\"$cluster|\\", }",\n "format": "time_series",\n "instant": true,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Ceph health status",\n "transparent": true,\n "type": "stat"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": null,\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "max": 1,\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(245, 54, 54, 0.9)"\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 0.10000000000000001\n },\n {\n "color": "rgba(50, 172, 45, 0.97)",\n "value": 0.29999999999999999\n }\n ]\n },\n "unit": "percentunit"\n }\n },\n "gridPos": {\n "h": 6,\n "w": 3,\n "x": 3,\n "y": 1\n },\n "id": 4,\n "interval": "1m",\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "showThresholdLabels": false,\n "showThresholdMarkers": true\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "(ceph_cluster_total_bytes{cluster=~\\"$cluster|\\", }-ceph_cluster_total_used_bytes{cluster=~\\"$cluster|\\", })/ceph_cluster_total_bytes{cluster=~\\"$cluster|\\", }",\n "format": "time_series",\n "instant": true,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Available Capacity",\n "transparent": false,\n "type": "gauge"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 2,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": null,\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(50, 172, 45, 0.97)"\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 0.025000000000000001\n },\n {\n "color": "rgba(245, 54, 54, 0.9)",\n "value": 1\n }\n ]\n },\n "unit": "bytes"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 6,\n "y": 1\n },\n "id": 5,\n "interval": "1m",\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "ceph_cluster_total_bytes{cluster=~\\"$cluster|\\", }",\n "format": "time_series",\n "instant": true,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Cluster Capacity",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 1,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": null,\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n },\n "unit": "Bps"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 9,\n "y": 1\n },\n "id": 6,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(rate(ceph_osd_op_w_in_bytes{cluster=~\\"$cluster|\\", }[$__rate_interval]))",\n "format": "time_series",\n "instant": true,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "Write Throughput",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 1,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": null,\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "#d44a3a"\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 0\n },\n {\n "color": "#9ac48a",\n "value": 0\n }\n ]\n },\n "unit": "Bps"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 12,\n "y": 1\n },\n "id": 7,\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(rate(ceph_osd_op_r_out_bytes{cluster=~\\"$cluster|\\", }[$__rate_interval]))",\n "format": "time_series",\n "instant": true,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "Read Throughput",\n "transparent": false,\n "type": "stat"\n },\n {\n "colorMode": "Panel",\n "colors": {\n "crit": "rgb(255, 0, 0)",\n "disable": "rgba(128, 128, 128, 0.9)",\n "ok": "rgba(50, 128, 45, 0.9)",\n "warn": "rgba(237, 129, 40, 0.9)"\n },\n "cornerRadius": 0,\n "datasource": "$datasource",\n "description": "",\n "displayName": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "All"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Out"\n },\n "properties": [\n {\n "id": "thresholds",\n "value": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 1\n }\n ]\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Down"\n },\n "properties": [\n {\n "id": "thresholds",\n "value": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 1\n }\n ]\n }\n }\n ]\n }\n ]\n },\n "flipCard": false,\n "flipTime": 5,\n "fontFormat": "Regular",\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 15,\n "y": 1\n },\n "id": 8,\n "isAutoScrollOnOverflow": false,\n "isGrayOnNoData": false,\n "isHideAlertsOnDisable": false,\n "isIgnoreOKColors": false,\n "links": [ ],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "aggregation": "Last",\n "alias": "All",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ceph_osd_metadata{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "All",\n "refId": "A",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "In",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_osd_in{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "In",\n "refId": "B",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "Out",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Warning / Critical",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_osd_in{cluster=~\\"$cluster|\\", } == bool 0)",\n "format": "time_series",\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Out",\n "refId": "C",\n "units": "none",\n "valueHandler": "Number Threshold",\n "warn": 1\n },\n {\n "aggregation": "Last",\n "alias": "Up",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_osd_up{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Up",\n "refId": "D",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "Down",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Warning / Critical",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_osd_up{cluster=~\\"$cluster|\\", } == bool 0)",\n "format": "time_series",\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Down",\n "refId": "E",\n "units": "none",\n "valueHandler": "Number Threshold",\n "warn": 1\n }\n ],\n "title": "OSDs",\n "transparent": false,\n "type": "stat"\n },\n {\n "colorMode": "Panel",\n "colors": {\n "crit": "rgba(245, 54, 54, 0.9)",\n "disable": "rgba(128, 128, 128, 0.9)",\n "ok": "rgba(50, 128, 45, 0.9)",\n "warn": "rgba(237, 129, 40, 0.9)"\n },\n "cornerRadius": 1,\n "datasource": "$datasource",\n "description": "",\n "displayName": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "none"\n }\n },\n "flipCard": false,\n "flipTime": 5,\n "fontFormat": "Regular",\n "gridPos": {\n "h": 6,\n "w": 3,\n "x": 21,\n "y": 1\n },\n "id": 9,\n "isAutoScrollOnOverflow": false,\n "isGrayOnNoData": false,\n "isHideAlertsOnDisable": false,\n "isIgnoreOKColors": false,\n "links": [ ],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "aggregation": "Last",\n "alias": "Active",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ceph_mgr_status{cluster=~\\"$cluster|\\", } == 1) or vector(0)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Active",\n "refId": "A",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "Standby",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ceph_mgr_status{cluster=~\\"$cluster|\\", } == 0) or vector(0)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Standby",\n "refId": "B",\n "units": "none",\n "valueHandler": "Number Threshold"\n }\n ],\n "title": "MGRs",\n "transparent": false,\n "type": "stat"\n },\n {\n "colorMode": "Panel",\n "colors": {\n "crit": "rgba(245, 54, 54, 0.9)",\n "disable": "rgba(128, 128, 128, 0.9)",\n "ok": "rgba(50, 128, 45, 0.9)",\n "warn": "rgba(237, 129, 40, 0.9)"\n },\n "cornerRadius": 1,\n "datasource": "$datasource",\n "description": "",\n "displayName": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 1\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "Critical"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "red",\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Warning"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "#987d24",\n "mode": "fixed"\n }\n }\n ]\n }\n ]\n },\n "flipCard": false,\n "flipTime": 5,\n "fontFormat": "Regular",\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 0,\n "y": 4\n },\n "id": 10,\n "isAutoScrollOnOverflow": false,\n "isGrayOnNoData": false,\n "isHideAlertsOnDisable": false,\n "isIgnoreOKColors": false,\n "links": [ ],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "aggregation": "Last",\n "alias": "Active",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ALERTS{alertstate=\\"firing\\",alertname=~\\"^Ceph.+\\", severity=\\"critical\\", cluster=~\\"$cluster|\\", }) OR vector(0)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Critical",\n "refId": "A",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "Standby",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ALERTS{alertstate=\\"firing\\",alertname=~\\"^Ceph.+\\", severity=\\"warning\\", cluster=~\\"$cluster|\\", }) OR vector(0)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Warning",\n "refId": "B",\n "units": "none",\n "valueHandler": "Number Threshold"\n }\n ],\n "title": "Firing Alerts",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "displayName": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "rgba(50, 172, 45, 0.97)",\n "value": null\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 0.025000000000000001\n },\n {\n "color": "rgba(245, 54, 54, 0.9)",\n "value": 0.10000000000000001\n }\n ]\n },\n "unit": "bytes"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 6,\n "y": 4\n },\n "id": 11,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "ceph_cluster_total_used_bytes{cluster=~\\"$cluster|\\", }",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "Used Capacity",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "displayName": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "ops"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 9,\n "y": 4\n },\n "id": 12,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(rate(ceph_osd_op_w{cluster=~\\"$cluster|\\", }[$__rate_interval]))",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "Write IOPS",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "displayName": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "result": {\n "text": "N/A"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "#d44a3a",\n "value": null\n },\n {\n "color": "rgba(237, 129, 40, 0.89)",\n "value": 0\n },\n {\n "color": "#9ac48a",\n "value": 0\n }\n ]\n },\n "unit": "ops"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 12,\n "y": 4\n },\n "id": 13,\n "links": [ ],\n "maxDataPoints": 100,\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(rate(ceph_osd_op_r{cluster=~\\"$cluster|\\", }[$__rate_interval]))",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "Read IOPS",\n "transparent": false,\n "type": "stat"\n },\n {\n "colorMode": "Panel",\n "colors": {\n "crit": "rgba(245, 54, 54, 0.9)",\n "disable": "rgba(128, 128, 128, 0.9)",\n "ok": "rgba(50, 128, 45, 0.9)",\n "warn": "rgba(237, 129, 40, 0.9)"\n },\n "cornerRadius": 1,\n "datasource": "$datasource",\n "description": "",\n "displayName": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "none"\n }\n },\n "flipCard": false,\n "flipTime": 5,\n "fontFormat": "Regular",\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 15,\n "y": 4\n },\n "id": 14,\n "isAutoScrollOnOverflow": false,\n "isGrayOnNoData": false,\n "isHideAlertsOnDisable": false,\n "isIgnoreOKColors": false,\n "links": [ ],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "aggregation": "Last",\n "alias": "In Quorum",\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_mon_quorum_status{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "In Quorum",\n "refId": "A",\n "units": "none",\n "valueHandler": "Text Only"\n },\n {\n "aggregation": "Last",\n "alias": "Total",\n "crit": 1,\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ceph_mon_quorum_status{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Total",\n "refId": "B",\n "units": "none",\n "valueHandler": "Text Only",\n "warn": 2\n },\n {\n "aggregation": "Last",\n "alias": "MONs out of Quorum",\n "crit": 1.6000000000000001,\n "datasource": "$datasource",\n "decimals": 2,\n "displayAliasType": "Warning / Critical",\n "displayType": "Annotation",\n "displayValueWithAlias": "Never",\n "expr": "count(ceph_mon_quorum_status{cluster=~\\"$cluster|\\", }) - sum(ceph_mon_quorum_status{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "MONs out of Quorum",\n "range": true,\n "refId": "C",\n "units": "none",\n "valueHandler": "Number Threshold",\n "warn": 1.1000000000000001\n }\n ],\n "title": "Monitors",\n "transparent": false,\n "type": "stat"\n },\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 7\n },\n "id": 15,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "CLUSTER STATS",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "datasource": {\n "type": "datasource",\n "uid": "grafana"\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 0,\n "y": 8\n },\n "id": 16,\n "limit": 10,\n "onlyAlertsOnDashboard": true,\n "options": {\n "alertInstanceLabelFilter": "{alertname=~\\"^Ceph.+\\", cluster=~\\"$cluster|\\", }",\n "alertName": "",\n "dashboardAlerts": false,\n "groupBy": [ ],\n "groupMode": "default",\n "maxItems": 20,\n "sortOrder": 1,\n "stateFilter": {\n "error": true,\n "firing": true,\n "noData": false,\n "normal": false,\n "pending": true\n },\n "viewMode": "list"\n },\n "show": "current",\n "sortOrder": 1,\n "stateFilter": [ ],\n "title": "Alerts",\n "type": "alertlist"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 40,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 0,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "thresholds": {\n "mode": "percentage",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "#c0921f",\n "value": 75\n },\n {\n "color": "#E02F44",\n "value": 85\n }\n ]\n },\n "unit": "bytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "Total Capacity"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "red",\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Used"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "green",\n "mode": "fixed"\n }\n },\n {\n "id": "custom.thresholdsStyle",\n "value": {\n "mode": "dashed"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 8,\n "y": 8\n },\n "id": 17,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [\n "last"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "sortBy": "Last",\n "sortDesc": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "ceph_cluster_total_bytes{cluster=~\\"$cluster|\\", }",\n "format": "time_series",\n "instant": false,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Total Capacity",\n "range": true,\n "refId": "A",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "ceph_cluster_total_used_bytes{cluster=~\\"$cluster|\\", }",\n "format": "time_series",\n "instant": false,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Used",\n "range": true,\n "refId": "B",\n "step": 300\n }\n ],\n "title": "Capacity",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 85\n }\n ]\n },\n "unit": "bytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 16,\n "y": 8\n },\n "id": 18,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [\n "mean",\n "lastNotNull",\n "max",\n "min"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(rate(ceph_osd_op_w_in_bytes{cluster=~\\"$cluster|\\", }[$__rate_interval]))",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Write",\n "range": true,\n "refId": "A",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "sum(rate(ceph_osd_op_r_out_bytes{cluster=~\\"$cluster|\\", }[$__rate_interval]))",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Read",\n "range": true,\n "refId": "B",\n "step": 300\n }\n ],\n "title": "Cluster Throughput",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "bytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 0,\n "y": 16\n },\n "id": 19,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [\n "mean",\n "lastNotNull",\n "max",\n "min"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(rate(ceph_osd_op_w{cluster=~\\"$cluster|\\", }[$__rate_interval]))",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Write",\n "range": true,\n "refId": "A",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "sum(rate(ceph_osd_op_r{cluster=~\\"$cluster|\\", }[$__rate_interval]))",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Read",\n "range": true,\n "refId": "B",\n "step": 300\n }\n ],\n "title": "IOPS",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "bytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 8,\n "y": 16\n },\n "id": 20,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "right",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "(ceph_pool_bytes_used{cluster=~\\"$cluster|\\", }) *on (pool_id) group_left(name)(ceph_pool_metadata{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Pool Used Bytes",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "bytes"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "rbd Stored"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "transparent",\n "mode": "fixed"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 16,\n "y": 16\n },\n "id": 21,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "table",\n "placement": "right",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "(ceph_pool_stored_raw{cluster=~\\"$cluster|\\", }) *on (pool_id) group_left(name)(ceph_pool_metadata{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "hide": false,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "range": true,\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Pool Used RAW Bytes",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 0,\n "y": 24\n },\n "id": 22,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "(ceph_pool_quota_objects{cluster=~\\"$cluster|\\", }) *on (pool_id) group_left(name)(ceph_pool_metadata{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Pool Objects Quota",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "bytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 8,\n "y": 24\n },\n "id": 23,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "(ceph_pool_quota_bytes{cluster=~\\"$cluster|\\", }) *on (pool_id) group_left(name)(ceph_pool_metadata{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Pool Quota Bytes",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 16,\n "y": 24\n },\n "id": 24,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "right",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "(ceph_pool_objects{cluster=~\\"$cluster|\\", }) * on (pool_id) group_left(name)(ceph_pool_metadata{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "{{name}}",\n "refId": "A"\n }\n ],\n "title": "Objects Per Pool",\n "type": "timeseries"\n },\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 31\n },\n "id": 25,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "OBJECTS",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 2,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byRegexp",\n "options": "/^Total.*$/"\n },\n "properties": [\n {\n "id": "custom.stacking",\n "value": {\n "group": false,\n "mode": "normal"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 12,\n "w": 6,\n "x": 0,\n "y": 32\n },\n "id": 26,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": false\n },\n "tooltip": {\n "mode": "multi",\n "sort": "asc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pool_objects{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Total",\n "range": true,\n "refId": "A",\n "step": 200\n }\n ],\n "title": "OSD Type Count",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 2,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byRegexp",\n "options": "/^Total.*$/"\n },\n "properties": [\n {\n "id": "custom.stacking",\n "value": {\n "group": false,\n "mode": "normal"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 12,\n "w": 8,\n "x": 6,\n "y": 32\n },\n "id": 27,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "right",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "asc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_active{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Active",\n "range": true,\n "refId": "A"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_clean{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Clean",\n "range": true,\n "refId": "B"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_peering{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Peering",\n "range": true,\n "refId": "C"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_degraded{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Degraded",\n "range": true,\n "refId": "D",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_stale{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Stale",\n "range": true,\n "refId": "E",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_unclean_pgs{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Unclean",\n "range": true,\n "refId": "F",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_undersized{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Undersized",\n "range": true,\n "refId": "G",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_incomplete{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Incomplete",\n "range": true,\n "refId": "H"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_forced_backfill{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Forced Backfill",\n "range": true,\n "refId": "I"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_forced_recovery{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Forced Recovery",\n "range": true,\n "refId": "J"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_creating{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Creating",\n "range": true,\n "refId": "K"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_wait_backfill{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Wait Backfill",\n "range": true,\n "refId": "L"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_deep{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Deep",\n "range": true,\n "refId": "M"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_scrubbing{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Scrubbing",\n "range": true,\n "refId": "N"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_recovering{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Recovering",\n "range": true,\n "refId": "O"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_repair{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Repair",\n "range": true,\n "refId": "P"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_down{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Down",\n "range": true,\n "refId": "Q"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_peered{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Peered",\n "range": true,\n "refId": "R"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_backfill{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Backfill",\n "range": true,\n "refId": "S"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_remapped{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Remapped",\n "range": true,\n "refId": "T"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_backfill_toofull{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Backfill Toofull",\n "range": true,\n "refId": "U"\n }\n ],\n "title": "PGs State",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 2,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byRegexp",\n "options": "/^Total.*$/"\n },\n "properties": [\n {\n "id": "custom.stacking",\n "value": {\n "group": false,\n "mode": "normal"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 6,\n "w": 10,\n "x": 14,\n "y": 32\n },\n "id": 28,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [\n "mean",\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "right",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "asc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_degraded{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Degraded",\n "range": true,\n "refId": "A",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_stale{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Stale",\n "range": true,\n "refId": "B",\n "step": 300\n },\n {\n "datasource": "$datasource",\n "expr": "sum(ceph_pg_undersized{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "Undersized",\n "range": true,\n "refId": "C",\n "step": 300\n }\n ],\n "title": "Stuck PGs",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 2,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 6,\n "w": 10,\n "x": 14,\n "y": 38\n },\n "id": 29,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": false\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(rate(ceph_osd_recovery_ops{cluster=~\\"$cluster|\\", }[$__rate_interval]))",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "OPS",\n "refId": "A",\n "step": 300\n }\n ],\n "title": "Recovery Operations",\n "type": "timeseries"\n },\n {\n "collapse": false,\n "collapsed": true,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 44\n },\n "id": 30,\n "panels": [\n {\n "cards": {\n "cardPadding": null,\n "cardRound": null\n },\n "color": {\n "cardColor": "#b4ff00",\n "colorScale": "sqrt",\n "colorScheme": "interpolateOranges",\n "exponent": 0.5,\n "mode": "opacity"\n },\n "dataFormat": "timeseries",\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "scaleDistribution": {\n "type": "linear"\n }\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 0,\n "y": 42\n },\n "heatmap": { },\n "hideZeroBuckets": false,\n "highlightCards": true,\n "id": 31,\n "legend": {\n "show": true\n },\n "options": {\n "calculate": true,\n "calculation": {\n "yBuckets": {\n "mode": "count",\n "scale": {\n "log": 2,\n "type": "log"\n },\n "value": "1"\n }\n },\n "cellGap": 2,\n "cellValues": { },\n "color": {\n "exponent": 0.5,\n "fill": "#b4ff00",\n "mode": "opacity",\n "reverse": false,\n "scale": "exponential",\n "scheme": "Oranges",\n "steps": 128\n },\n "exemplars": {\n "color": "rgba(255,0,255,0.7)"\n },\n "filterValues": {\n "le": 1.0000000000000001e-09\n },\n "legend": {\n "show": true\n },\n "rowsFrame": {\n "layout": "auto"\n },\n "showValue": "never",\n "tooltip": {\n "show": true,\n "yHistogram": false\n },\n "yAxis": {\n "axisPlacement": "left",\n "min": "0",\n "reverse": false,\n "unit": "ms"\n }\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "ceph_osd_apply_latency_ms{cluster=~\\"$cluster|\\", }",\n "format": "time_series",\n "instant": false,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "OSD Apply Latency Distribution",\n "tooltip": {\n "show": true,\n "showHistogram": false\n },\n "type": "heatmap",\n "xAxis": {\n "show": true\n },\n "xBucketNumber": null,\n "xBucketSize": "",\n "yAxis": {\n "decimals": null,\n "format": "ms",\n "logBase": 2,\n "max": null,\n "min": "0",\n "show": true,\n "splitFactor": 1\n },\n "yBucketBound": "auto",\n "yBucketNumber": null,\n "yBucketSize": 10\n },\n {\n "cards": {\n "cardPadding": null,\n "cardRound": null\n },\n "color": {\n "cardColor": "#65c5db",\n "colorScale": "sqrt",\n "colorScheme": "interpolateOranges",\n "exponent": 0.5,\n "mode": "opacity"\n },\n "dataFormat": "timeseries",\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "scaleDistribution": {\n "type": "linear"\n }\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 12,\n "y": 42\n },\n "heatmap": { },\n "hideZeroBuckets": false,\n "highlightCards": true,\n "id": 32,\n "legend": {\n "show": true\n },\n "options": {\n "calculate": true,\n "calculation": {\n "yBuckets": {\n "mode": "count",\n "scale": {\n "log": 2,\n "type": "log"\n }\n }\n },\n "cellGap": 2,\n "cellValues": { },\n "color": {\n "exponent": 0.5,\n "fill": "#65c5db",\n "mode": "opacity",\n "reverse": false,\n "scale": "exponential",\n "scheme": "Oranges",\n "steps": 128\n },\n "exemplars": {\n "color": "rgba(255,0,255,0.7)"\n },\n "filterValues": {\n "le": 1.0000000000000001e-09\n },\n "legend": {\n "show": true\n },\n "rowsFrame": {\n "layout": "auto"\n },\n "showValue": "never",\n "tooltip": {\n "show": true,\n "yHistogram": false\n },\n "yAxis": {\n "axisPlacement": "left",\n "min": "0",\n "reverse": false,\n "unit": "ms"\n }\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "ceph_osd_commit_latency_ms{cluster=~\\"$cluster|\\", }",\n "format": "time_series",\n "instant": false,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "OSD Commit Latency Distribution",\n "tooltip": {\n "show": true,\n "showHistogram": false\n },\n "type": "heatmap",\n "xAxis": {\n "show": true\n },\n "xBucketNumber": null,\n "xBucketSize": "",\n "yAxis": {\n "decimals": null,\n "format": "ms",\n "logBase": 2,\n "max": null,\n "min": "0",\n "show": true,\n "splitFactor": 1\n },\n "yBucketBound": "auto",\n "yBucketNumber": null,\n "yBucketSize": 10\n },\n {\n "cards": {\n "cardPadding": null,\n "cardRound": null\n },\n "color": {\n "cardColor": "#806eb7",\n "colorScale": "sqrt",\n "colorScheme": "interpolateOranges",\n "exponent": 0.5,\n "mode": "opacity"\n },\n "dataFormat": "timeseries",\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "scaleDistribution": {\n "type": "linear"\n }\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 0,\n "y": 50\n },\n "heatmap": { },\n "hideZeroBuckets": false,\n "highlightCards": true,\n "id": 33,\n "legend": {\n "show": true\n },\n "options": {\n "calculate": true,\n "calculation": {\n "yBuckets": {\n "mode": "count",\n "scale": {\n "log": 2,\n "type": "log"\n }\n }\n },\n "cellGap": 2,\n "cellValues": { },\n "color": {\n "exponent": 0.5,\n "fill": "#806eb7",\n "mode": "opacity",\n "reverse": false,\n "scale": "exponential",\n "scheme": "Oranges",\n "steps": 128\n },\n "exemplars": {\n "color": "rgba(255,0,255,0.7)"\n },\n "filterValues": {\n "le": 1.0000000000000001e-09\n },\n "legend": {\n "show": true\n },\n "rowsFrame": {\n "layout": "auto"\n },\n "showValue": "never",\n "tooltip": {\n "show": true,\n "yHistogram": false\n },\n "yAxis": {\n "axisPlacement": "left",\n "decimals": 2,\n "min": "0",\n "reverse": false,\n "unit": "ms"\n }\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "rate(ceph_osd_op_r_latency_sum{cluster=~\\"$cluster|\\", }[$__rate_interval]) / rate(ceph_osd_op_r_latency_count{cluster=~\\"$cluster|\\", }[$__rate_interval]) >= 0",\n "format": "time_series",\n "instant": false,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "OSD Read Op Latency Distribution",\n "tooltip": {\n "show": true,\n "showHistogram": false\n },\n "type": "heatmap",\n "xAxis": {\n "show": true\n },\n "xBucketNumber": null,\n "xBucketSize": "",\n "yAxis": {\n "decimals": null,\n "format": "ms",\n "logBase": 2,\n "max": null,\n "min": "0",\n "show": true,\n "splitFactor": 1\n },\n "yBucketBound": "auto",\n "yBucketNumber": null,\n "yBucketSize": null\n },\n {\n "cards": {\n "cardPadding": null,\n "cardRound": null\n },\n "color": {\n "cardColor": "#f9934e",\n "colorScale": "sqrt",\n "colorScheme": "interpolateOranges",\n "exponent": 0.5,\n "mode": "opacity"\n },\n "dataFormat": "timeseries",\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "scaleDistribution": {\n "type": "linear"\n }\n }\n }\n },\n "gridPos": {\n "h": 8,\n "w": 12,\n "x": 12,\n "y": 50\n },\n "heatmap": { },\n "hideZeroBuckets": false,\n "highlightCards": true,\n "id": 34,\n "legend": {\n "show": true\n },\n "options": {\n "calculate": true,\n "calculation": {\n "yBuckets": {\n "mode": "count",\n "scale": {\n "log": 2,\n "type": "log"\n }\n }\n },\n "cellGap": 2,\n "cellValues": { },\n "color": {\n "exponent": 0.5,\n "fill": "#f9934e",\n "mode": "opacity",\n "reverse": false,\n "scale": "exponential",\n "scheme": "Oranges",\n "steps": 128\n },\n "exemplars": {\n "color": "rgba(255,0,255,0.7)"\n },\n "filterValues": {\n "le": 1.0000000000000001e-09\n },\n "legend": {\n "show": true\n },\n "rowsFrame": {\n "layout": "auto"\n },\n "showValue": "never",\n "tooltip": {\n "show": true,\n "yHistogram": false\n },\n "yAxis": {\n "axisPlacement": "left",\n "decimals": 2,\n "min": "0",\n "reverse": false,\n "unit": "ms"\n }\n },\n "pluginVersion": "9.4.7",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "rate(ceph_osd_op_w_latency_sum{cluster=~\\"$cluster|\\", }[$__rate_interval]) / rate(ceph_osd_op_w_latency_count{cluster=~\\"$cluster|\\", }[$__rate_interval]) >= 0",\n "format": "time_series",\n "instant": false,\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "title": "OSD Write Op Latency Distribution",\n "tooltip": {\n "show": true,\n "showHistogram": false\n },\n "type": "heatmap",\n "xAxis": {\n "show": true\n },\n "xBucketNumber": null,\n "xBucketSize": "",\n "yAxis": {\n "decimals": null,\n "format": "ms",\n "logBase": 2,\n "max": null,\n "min": "0",\n "show": true,\n "splitFactor": 1\n },\n "yBucketBound": "auto",\n "yBucketNumber": null,\n "yBucketSize": null\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "ms"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 12,\n "x": 0,\n "y": 58\n },\n "id": 35,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "avg(rate(ceph_osd_op_r_latency_sum{cluster=~\\"$cluster|\\", }[$__rate_interval]) / rate(ceph_osd_op_r_latency_count{cluster=~\\"$cluster|\\", }[$__rate_interval]) >= 0)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Read",\n "refId": "A"\n },\n {\n "datasource": "$datasource",\n "expr": "avg(rate(ceph_osd_op_w_latency_sum{cluster=~\\"$cluster|\\", }[$__rate_interval]) / rate(ceph_osd_op_w_latency_count{cluster=~\\"$cluster|\\", }[$__rate_interval]) >= 0)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Write",\n "refId": "B"\n }\n ],\n "title": "Recovery Operations",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 10,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "ms"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 12,\n "x": 12,\n "y": 58\n },\n "id": 36,\n "interval": "$interval",\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull",\n "max"\n ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "avg(ceph_osd_apply_latency_ms{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "apply",\n "metric": "ceph_osd_perf_apply_latency_seconds",\n "refId": "A",\n "step": 4\n },\n {\n "datasource": "$datasource",\n "expr": "avg(ceph_osd_commit_latency_ms{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "interval": "$interval",\n "intervalFactor": 1,\n "legendFormat": "commit",\n "metric": "ceph_osd_perf_commit_latency_seconds",\n "refId": "B",\n "step": 4\n }\n ],\n "title": "AVG OSD Apply + Commit Latency",\n "type": "timeseries"\n }\n ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "LATENCY",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "collapse": true,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 45\n },\n "id": 37,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "columns": [ ],\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "left",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": false,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "Time"\n },\n "properties": [\n {\n "id": "custom.hidden",\n "value": true\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 6,\n "w": 24,\n "x": 0,\n "y": 46\n },\n "id": 38,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "9.4.7",\n "styles": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "exemplar": false,\n "expr": "count by (ceph_version)(ceph_osd_metadata{cluster=~\\"$cluster|\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "OSD Services",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": "$datasource",\n "exemplar": false,\n "expr": "count by (ceph_version)(ceph_mon_metadata{cluster=~\\"$cluster|\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Mon Services",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": "$datasource",\n "exemplar": false,\n "expr": "count by (ceph_version)(ceph_mds_metadata{cluster=~\\"$cluster|\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "MDS Services",\n "range": false,\n "refId": "C"\n },\n {\n "datasource": "$datasource",\n "exemplar": false,\n "expr": "count by (ceph_version)(ceph_rgw_metadata{cluster=~\\"$cluster|\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "RGW Services",\n "range": false,\n "refId": "D"\n },\n {\n "datasource": "$datasource",\n "exemplar": false,\n "expr": "count by (ceph_version)(ceph_mgr_metadata{cluster=~\\"$cluster|\\", })",\n "format": "table",\n "hide": false,\n "instant": true,\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "MGR Services",\n "range": false,\n "refId": "E"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Ceph Versions",\n "transformations": [\n {\n "id": "merge",\n "options": { }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": { },\n "indexByName": { },\n "renameByName": {\n "Time": "",\n "Value #A": "OSD Services",\n "Value #B": "Mon Services",\n "Value #C": "MDS Services",\n "Value #D": "RGW Services",\n "Value #E": "MGR Services",\n "ceph_version": "Ceph Version"\n }\n }\n }\n ],\n "type": "table"\n }\n ],\n "refresh": "1m",\n "rows": [ ],\n "schemaVersion": 38,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "auto": true,\n "auto_count": 10,\n "auto_min": "1m",\n "current": {\n "text": "$__auto_interval_interval",\n "value": "$__auto_interval_interval"\n },\n "hide": 0,\n "label": "Interval",\n "name": "interval",\n "options": [\n {\n "selected": true,\n "text": "auto",\n "value": "$__auto_interval_interval"\n },\n {\n "selected": false,\n "text": "5s",\n "value": "5s"\n },\n {\n "selected": false,\n "text": "10s",\n "value": "10s"\n },\n {\n "selected": false,\n "text": "30s",\n "value": "30s"\n },\n {\n "selected": false,\n "text": "1m",\n "value": "1m"\n },\n {\n "selected": false,\n "text": "10m",\n "value": "10m"\n },\n {\n "selected": false,\n "text": "30m",\n "value": "30m"\n },\n {\n "selected": false,\n "text": "1h",\n "value": "1h"\n },\n {\n "selected": false,\n "text": "6h",\n "value": "6h"\n },\n {\n "selected": false,\n "text": "12h",\n "value": "12h"\n },\n {\n "selected": false,\n "text": "1d",\n "value": "1d"\n },\n {\n "selected": false,\n "text": "7d",\n "value": "7d"\n },\n {\n "selected": false,\n "text": "14d",\n "value": "14d"\n },\n {\n "selected": false,\n "text": "30d",\n "value": "30d"\n }\n ],\n "query": "5s,10s,30s,1m,10m,30m,1h,6h,12h,1d,7d,14d,30d",\n "refresh": 2,\n "type": "interval",\n "valuelabels": { }\n }\n ]\n },\n "time": {\n "from": "now-6h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "Ceph Cluster - Advanced",\n "uid": "dn13KBeTv",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/smb-overview.json': '{\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": {\n "type": "datasource",\n "uid": "grafana"\n },\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [],\n "type": "dashboard"\n }\n ]\n },\n "description": "SMB Overview dashboard shows data across all clusters and hosts associated with the SMB service.",\n "editable": false,\n "fiscalYearStartMonth": 0,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [],\n "panels": [\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "description": "SMB metrics daemon health.",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [],\n "mappings": [\n {\n "options": {\n "0": {\n "color": "red",\n "text": "Down"\n },\n "1": {\n "color": "green",\n "text": "Up"\n }\n },\n "type": "value"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "short"\n },\n "overrides": []\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "percentChangeColorMode": "standard",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "showPercentChange": false,\n "textMode": "auto",\n "wideLayout": true\n },\n "pluginVersion": "11.6.0",\n "targets": [\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "expr": "sum(smb_metrics_status{instance=~\\"$hostname\\"}) by (instance)",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Prometheus SMB metrics status",\n "type": "stat"\n },\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "description": "Number of nodes per cluster/s",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [],\n "mappings": [],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "none"\n },\n "overrides": []\n },\n "gridPos": {\n "h": 4,\n "w": 8,\n "x": 8,\n "y": 0\n },\n "id": 3,\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "percentChangeColorMode": "standard",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "showPercentChange": false,\n "textMode": "auto",\n "wideLayout": true\n },\n "pluginVersion": "11.6.0",\n "targets": [\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "expr": "sum(count by (netbiosname) (smb_sessions_total * on (instance) group_left (netbiosname) smb_metrics_status{netbiosname=~\\"$SMBcluster\\"}))",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Nodes",\n "type": "stat"\n },\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "description": "Number of users currently logged in Cluster/s",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [],\n "mappings": [],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "none"\n },\n "overrides": []\n },\n "gridPos": {\n "h": 4,\n "w": 8,\n "x": 16,\n "y": 0\n },\n "id": 4,\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "percentChangeColorMode": "standard",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "showPercentChange": false,\n "textMode": "auto",\n "wideLayout": true\n },\n "pluginVersion": "11.6.0",\n "targets": [\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "exemplar": false,\n "expr": "sum(sum by(netbiosname)(smb_sessions_total * on (instance) group_left (netbiosname) smb_metrics_status{netbiosname=~\\"$SMBcluster\\"}) / (count by ( netbiosname)(smb_sessions_total * on (instance) group_left (netbiosname) smb_metrics_status{netbiosname=~\\"$SMBcluster\\"})))",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Active sessions",\n "type": "stat"\n },\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "description": "Number of remote machines using a share in cluster/s",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [],\n "mappings": [],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "none"\n },\n "overrides": []\n },\n "gridPos": {\n "h": 4,\n "w": 8,\n "x": 8,\n "y": 4\n },\n "id": 5,\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "percentChangeColorMode": "standard",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "showPercentChange": false,\n "textMode": "auto",\n "wideLayout": true\n },\n "pluginVersion": "11.6.0",\n "targets": [\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "exemplar": false,\n "expr": "sum(sum by(netbiosname)(smb_share_activity * on (instance) group_left (netbiosname) smb_metrics_status{netbiosname=~\\"$SMBcluster\\"}) / (count by ( netbiosname)(smb_share_activity * on (instance) group_left (netbiosname) smb_metrics_status{netbiosname=~\\"$SMBcluster\\"})))",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Shares activity",\n "type": "stat"\n },\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "description": "Number of currently active SMB user per cluster/s",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [],\n "mappings": [],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "none"\n },\n "overrides": []\n },\n "gridPos": {\n "h": 4,\n "w": 8,\n "x": 16,\n "y": 4\n },\n "id": 6,\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "percentChangeColorMode": "standard",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "showPercentChange": false,\n "textMode": "auto",\n "wideLayout": true\n },\n "pluginVersion": "11.6.0",\n "targets": [\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "exemplar": false,\n "expr": "sum(sum by(netbiosname)(smb_users_total * on (instance) group_left (netbiosname) smb_metrics_status{netbiosname=~\\"$SMBcluster\\"}) / (count by ( netbiosname)(smb_users_total * on (instance) group_left (netbiosname) smb_metrics_status{netbiosname=~\\"$SMBcluster\\"})))",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Active users",\n "type": "stat"\n },\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "description": "Current total ingress throughput, bytes going in per second",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [],\n "mappings": [],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": []\n },\n "gridPos": {\n "h": 5,\n "w": 6,\n "x": 0,\n "y": 8\n },\n "id": 7,\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "percentChangeColorMode": "standard",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "showPercentChange": false,\n "textMode": "auto",\n "wideLayout": true\n },\n "pluginVersion": "11.6.0",\n "targets": [\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "exemplar": false,\n "expr": "sum(rate(smb_smb2_request_inbytes{instance=~\\"$hostname\\", share=~\\"$share\\", client=~\\"$client\\"}[$__rate_interval]))",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Ingress throughtput",\n "type": "stat"\n },\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "description": "Current total request time for the sum of all runing operations",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [],\n "mappings": [],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "µs"\n },\n "overrides": []\n },\n "gridPos": {\n "h": 5,\n "w": 6,\n "x": 6,\n "y": 8\n },\n "id": 8,\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "percentChangeColorMode": "standard",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "showPercentChange": false,\n "textMode": "auto",\n "wideLayout": true\n },\n "pluginVersion": "11.6.0",\n "targets": [\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "exemplar": false,\n "expr": "sum(rate(smb_smb2_request_duration_microseconds_sum{instance=~\\"$hostname\\", share=~\\"$share\\", client=~\\"$client\\"}[$__rate_interval]))",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Latency",\n "type": "stat"\n },\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "description": "Current total egress throughput, bytes going out per second",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [],\n "mappings": [],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "decbytes"\n },\n "overrides": []\n },\n "gridPos": {\n "h": 5,\n "w": 6,\n "x": 12,\n "y": 8\n },\n "id": 9,\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "percentChangeColorMode": "standard",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "showPercentChange": false,\n "textMode": "auto",\n "wideLayout": true\n },\n "pluginVersion": "11.6.0",\n "targets": [\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "exemplar": false,\n "expr": "sum(rate(smb_smb2_request_outbytes{instance=~\\"$hostname\\", share=~\\"$share\\", client=~\\"$client\\"}[$__rate_interval]))",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Egress throughtput",\n "type": "stat"\n },\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "description": "Current total number of operations per second",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [],\n "mappings": [],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "short"\n },\n "overrides": []\n },\n "gridPos": {\n "h": 5,\n "w": 6,\n "x": 18,\n "y": 8\n },\n "id": 10,\n "options": {\n "colorMode": "none",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "percentChangeColorMode": "standard",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "showPercentChange": false,\n "textMode": "auto",\n "wideLayout": true\n },\n "pluginVersion": "11.6.0",\n "targets": [\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "exemplar": false,\n "expr": "sum(rate(smb_smb2_request_total{instance=~\\"$hostname\\", share=~\\"$share\\", client=~\\"$client\\"}[$__rate_interval]))",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "I/O",\n "type": "stat"\n },\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisBorderShow": false,\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "barWidthFactor": 0.6,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "insertNulls": false,\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "mappings": [],\n "thresholds": {\n "mode": "absolute",\n "steps": []\n },\n "unit": "bytes"\n },\n "overrides": []\n },\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 0,\n "y": 13\n },\n "id": 11,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "right",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "hideZeros": false,\n "mode": "single",\n "sort": "none"\n }\n },\n "pluginVersion": "11.6.0",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "expr": "sum by (operation) (rate(smb_smb2_request_inbytes{instance=~\\"$hostname\\", share=~\\"$share\\", client=~\\"$client\\"}[$__rate_interval]))",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "Input - {{operation}}",\n "range": true,\n "refId": "A"\n },\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "expr": "sum by (operation) (rate(smb_smb2_request_outbytes{instance=~\\"$hostname\\", share=~\\"$share\\", client=~\\"$client\\"}[$__rate_interval]))",\n "format": "time_series",\n "hide": false,\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "Output- {{operation}}",\n "range": true,\n "refId": "B"\n }\n ],\n "title": "Throughput per Operation",\n "type": "timeseries"\n },\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisBorderShow": false,\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "barWidthFactor": 0.6,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "insertNulls": false,\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "mappings": [],\n "thresholds": {\n "mode": "absolute",\n "steps": []\n },\n "unit": "none"\n },\n "overrides": []\n },\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 12,\n "y": 13\n },\n "id": 12,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "right",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "hideZeros": false,\n "mode": "single",\n "sort": "none"\n }\n },\n "pluginVersion": "11.6.0",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "expr": "sum by (operation) (rate(smb_smb2_request_total{instance=~\\"$hostname\\", share=~\\"$share\\", client=~\\"$client\\"}[$__rate_interval]))",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{operation}}",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "IO per Operation",\n "type": "timeseries"\n },\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisBorderShow": false,\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "barWidthFactor": 0.6,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "insertNulls": false,\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 2,\n "mappings": [],\n "thresholds": {\n "mode": "absolute",\n "steps": []\n },\n "unit": "µs"\n },\n "overrides": []\n },\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 0,\n "y": 22\n },\n "id": 13,\n "options": {\n "legend": {\n "calcs": [\n "lastNotNull"\n ],\n "displayMode": "table",\n "placement": "right",\n "showLegend": true,\n "sortBy": "Last *",\n "sortDesc": true\n },\n "tooltip": {\n "hideZeros": false,\n "mode": "single",\n "sort": "none"\n }\n },\n "pluginVersion": "11.6.0",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": {\n "uid": "${datasource}"\n },\n "editorMode": "code",\n "expr": "sum by (operation) (rate(smb_smb2_request_duration_microseconds_sum{instance=~\\"$hostname\\", share=~\\"$share\\", client=~\\"$client\\"}[$__rate_interval]))",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "{{operation}}",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Latencies per operation",\n "type": "timeseries"\n }\n ],\n "preload": false,\n "refresh": "30s",\n "style": "dark",\n "schemaVersion": 41,\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "label": "Data Source",\n "name": "datasource",\n "options": [],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "current": {},\n "datasource": "$datasource",\n "includeAll": false,\n "name": "Cluster",\n "options": [],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 2,\n "regex": "",\n "type": "query"\n },\n {\n "current": {\n "text": "All",\n "value": "$__all"\n },\n "datasource": "$datasource",\n "includeAll": true,\n "label": "SMB Cluster",\n "name": "SMBcluster",\n "options": [],\n "query": "label_values(smb_metrics_status,netbiosname)",\n "refresh": 1,\n "sort": 1,\n "type": "query"\n },\n {\n "current": {\n "text": "All",\n "value": "$__all"\n },\n "datasource": "$datasource",\n "hide": 2,\n "includeAll": true,\n "label": "Hostname",\n "name": "hostname",\n "options": [],\n "query": "label_values(smb_metrics_status{netbiosname=~\\"$SMBcluster\\"},instance)",\n "refresh": 1,\n "sort": 1,\n "type": "query"\n },\n {\n "current": {\n "text": "All",\n "value": "$__all"\n },\n "definition": "label_values(smb_smb2_request_inbytes{netbiosname=~\\"$SMBcluster\\", instance=~\\"$hostname\\"},share)",\n "description": "",\n "includeAll": true,\n "label": "Share",\n "name": "share",\n "options": [],\n "query": {\n "qryType": 1,\n "query": "label_values(smb_smb2_request_inbytes{netbiosname=~\\"$SMBcluster\\", instance=~\\"$hostname\\"},share)",\n "refId": "PrometheusVariableQueryEditor-VariableQuery"\n },\n "refresh": 1,\n "regex": "",\n "type": "query"\n },\n {\n "current": {\n "text": "All",\n "value": "$__all"\n },\n "definition": "label_values(smb_smb2_request_inbytes{netbiosname=~\\"$SMBcluster\\", instance=~\\"$hostname\\", share=~\\"$share\\"},client)",\n "description": "",\n "includeAll": true,\n "label": "Client",\n "name": "client",\n "options": [],\n "query": {\n "qryType": 1,\n "query": "label_values(smb_smb2_request_inbytes{netbiosname=~\\"$SMBcluster\\", instance=~\\"$hostname\\", share=~\\"$share\\"},client)",\n "refId": "PrometheusVariableQueryEditor-VariableQuery"\n },\n "refresh": 1,\n "regex": "",\n "type": "query"\n }\n ]\n },\n "time": {\n "from": "now-6h",\n "to": "now"\n },\n "timepicker": {},\n "timezone": "",\n "title": "SMB Overview",\n "uid": "feem6ehrmi2o0b",\n "version": 1\n}\n', '/etc/grafana/provisioning/dashboards/osds-overview.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.0.0"\n },\n {\n "id": "grafana-piechart-panel",\n "name": "Pie Chart",\n "type": "panel",\n "version": "1.3.3"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n },\n {\n "id": "table",\n "name": "Table",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [\n {\n "asDropdown": true,\n "icon": "external link",\n "includeVars": true,\n "keepTime": true,\n "tags": [ ],\n "targetBlank": false,\n "title": "Browse Dashboards",\n "tooltip": "",\n "type": "dashboards",\n "url": ""\n }\n ],\n "panels": [\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "ms"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "avg(\\n rate(ceph_osd_op_r_latency_sum{cluster=~\\"$cluster|\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_r_latency_count{cluster=~\\"$cluster|\\", }[$__rate_interval]) * 1000\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "AVG read",\n "refId": "A"\n },\n {\n "expr": "max(\\n rate(ceph_osd_op_r_latency_sum{cluster=~\\"$cluster|\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_r_latency_count{cluster=~\\"$cluster|\\", }[$__rate_interval]) * 1000\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "MAX read",\n "refId": "B"\n },\n {\n "expr": "quantile(0.95,\\n (\\n rate(ceph_osd_op_r_latency_sum{cluster=~\\"$cluster|\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_r_latency_count{cluster=~\\"$cluster|\\", }[$__rate_interval]) * 1000\\n )\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "@95%ile",\n "refId": "C"\n }\n ],\n "title": "OSD Read Latencies",\n "type": "timeseries"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "null",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "ceph_daemon"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "OSD ID"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Latency (ms)"\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 4,\n "x": 8,\n "y": 0\n },\n "id": 3,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "10.4.0",\n "styles": [ ],\n "targets": [\n {\n "expr": "topk(10,\\n (sort(\\n (\\n rate(ceph_osd_op_r_latency_sum{cluster=~\\"$cluster|\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_r_latency_count{cluster=~\\"$cluster|\\", }[$__rate_interval]) *\\n 1000\\n )\\n ))\\n)\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Highest READ Latencies",\n "transformations": [\n {\n "id": "merge",\n "options": {\n "reducers": [ ]\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time": true,\n "cluster": true\n },\n "includeByName": { },\n "indexByName": { },\n "renameByName": { }\n }\n }\n ],\n "type": "table"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "ms"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 12,\n "y": 0\n },\n "id": 4,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "avg(\\n rate(ceph_osd_op_w_latency_sum{cluster=~\\"$cluster|\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_w_latency_count{cluster=~\\"$cluster|\\", }[$__rate_interval]) * 1000\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "AVG write",\n "refId": "A"\n },\n {\n "expr": "max(\\n rate(ceph_osd_op_w_latency_sum{cluster=~\\"$cluster|\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_w_latency_count{cluster=~\\"$cluster|\\", }[$__rate_interval]) * 1000\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "MAX write",\n "refId": "B"\n },\n {\n "expr": "quantile(0.95,\\n (\\n rate(ceph_osd_op_w_latency_sum{cluster=~\\"$cluster|\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_w_latency_count{cluster=~\\"$cluster|\\", }[$__rate_interval]) * 1000\\n )\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "@95%ile write",\n "refId": "C"\n }\n ],\n "title": "OSD Write Latencies",\n "type": "timeseries"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "description": "This table shows the osd\'s that are delivering the 10 highest write latencies within the cluster",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "null",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "ceph_daemon"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "OSD ID"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Latency (ms)"\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "mappings",\n "value": [\n {\n "options": {\n "NaN": {\n "index": 0,\n "text": "0.00"\n }\n },\n "type": "value"\n }\n ]\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 4,\n "x": 20,\n "y": 0\n },\n "id": 5,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "10.4.0",\n "styles": [ ],\n "targets": [\n {\n "expr": "topk(10,\\n (sort(\\n (rate(ceph_osd_op_w_latency_sum{cluster=~\\"$cluster|\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_w_latency_count{cluster=~\\"$cluster|\\", }[$__rate_interval]) *\\n 1000)\\n ))\\n)\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Highest WRITE Latencies",\n "transformations": [\n {\n "id": "merge",\n "options": {\n "reducers": [ ]\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time": true,\n "cluster": true\n },\n "includeByName": { },\n "indexByName": { },\n "renameByName": { }\n }\n }\n ],\n "type": "table"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n }\n },\n "mappings": [ ]\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 4,\n "x": 0,\n "y": 8\n },\n "id": 6,\n "options": {\n "displayLabels": [\n "percent"\n ],\n "legend": {\n "calcs": [ ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "values": [\n "percent",\n "value"\n ]\n },\n "pieType": "pie",\n "reduceOptions": { },\n "tooltip": {\n "mode": "single",\n "sort": "none"\n }\n },\n "targets": [\n {\n "expr": "count by (device_class) (ceph_osd_metadata{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device_class}}",\n "refId": "A"\n }\n ],\n "title": "OSD Types Summary",\n "type": "piechart"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n }\n },\n "mappings": [ ]\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 4,\n "x": 4,\n "y": 8\n },\n "id": 7,\n "options": {\n "displayLabels": [\n "percent"\n ],\n "legend": {\n "calcs": [ ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "values": [\n "percent",\n "value"\n ]\n },\n "pieType": "pie",\n "reduceOptions": { },\n "tooltip": {\n "mode": "single",\n "sort": "none"\n }\n },\n "targets": [\n {\n "expr": "count(ceph_bluefs_wal_total_bytes{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "bluestore",\n "refId": "A"\n },\n {\n "expr": "absent(ceph_bluefs_wal_total_bytes{cluster=~\\"$cluster|\\", }) * count(ceph_osd_metadata{cluster=~\\"$cluster|\\", })",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "filestore",\n "refId": "B"\n }\n ],\n "title": "OSD Objectstore Types",\n "type": "piechart"\n },\n {\n "datasource": "$datasource",\n "description": "The pie chart shows the various OSD sizes used within the cluster",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n }\n },\n "mappings": [ ]\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 4,\n "x": 8,\n "y": 8\n },\n "id": 8,\n "options": {\n "displayLabels": [\n "percent"\n ],\n "legend": {\n "calcs": [ ],\n "displayMode": "table",\n "placement": "bottom",\n "showLegend": true,\n "values": [\n "percent",\n "value"\n ]\n },\n "pieType": "pie",\n "reduceOptions": { },\n "tooltip": {\n "mode": "single",\n "sort": "none"\n }\n },\n "targets": [\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster|\\", } < 1099511627776)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<1TB",\n "refId": "A"\n },\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster|\\", } >= 1099511627776 < 2199023255552)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<2TB",\n "refId": "B"\n },\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster|\\", } >= 2199023255552 < 3298534883328)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<3TB",\n "refId": "C"\n },\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster|\\", } >= 3298534883328 < 4398046511104)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<4TB",\n "refId": "D"\n },\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster|\\", } >= 4398046511104 < 6597069766656)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<6TB",\n "refId": "E"\n },\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster|\\", } >= 6597069766656 < 8796093022208)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<8TB",\n "refId": "F"\n },\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster|\\", } >= 8796093022208 < 10995116277760)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<10TB",\n "refId": "G"\n },\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster|\\", } >= 10995116277760 < 13194139533312)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<12TB",\n "refId": "H"\n },\n {\n "expr": "count(ceph_osd_stat_bytes{cluster=~\\"$cluster|\\", } >= 13194139533312)",\n "format": "time_series",\n "intervalFactor": 2,\n "legendFormat": "<12TB+",\n "refId": "I"\n }\n ],\n "title": "OSD Size Summary",\n "type": "piechart"\n },\n {\n "aliasColors": { },\n "bars": true,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 12,\n "y": 8\n },\n "id": 9,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "ceph_osd_numpg{cluster=~\\"$cluster|\\", }",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "PGs per OSD",\n "refId": "A"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Distribution of PGs per OSD",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": 20,\n "mode": "histogram",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": "# of OSDs",\n "logBase": 1,\n "max": null,\n "min": "0",\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": "0",\n "show": true\n }\n ]\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": true,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "This gauge panel shows onode Hits ratio to help determine if increasing RAM per OSD could help improve the performance of the cluster",\n "format": "percentunit",\n "gauge": {\n "maxValue": 1,\n "minValue": 0,\n "show": true,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 8,\n "w": 4,\n "x": 20,\n "y": 8\n },\n "id": 10,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum(ceph_bluestore_onode_hits{cluster=~\\"$cluster|\\", }) / (\\n sum(ceph_bluestore_onode_hits{cluster=~\\"$cluster|\\", }) +\\n sum(ceph_bluestore_onode_misses{cluster=~\\"$cluster|\\", })\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": ".75",\n "title": "OSD onode Hits Ratio",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 16\n },\n "id": 11,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "R/W Profile",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 24,\n "x": 0,\n "y": 17\n },\n "id": 12,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "round(sum(rate(ceph_pool_rd{cluster=~\\"$cluster|\\", }[$__rate_interval])))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Reads",\n "refId": "A"\n },\n {\n "expr": "round(sum(rate(ceph_pool_wr{cluster=~\\"$cluster|\\", }[$__rate_interval])))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Writes",\n "refId": "B"\n }\n ],\n "title": "Read/Write Profile",\n "type": "timeseries"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "description": "This table shows the 10 OSDs with the highest number of slow ops",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "null",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "ceph_daemon"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "OSD ID"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Slow Ops"\n },\n {\n "id": "unit",\n "value": "none"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 5,\n "x": 0,\n "y": 25\n },\n "id": 13,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "10.4.0",\n "styles": [ ],\n "targets": [\n {\n "expr": "topk(10,\\n (ceph_daemon_health_metrics{type=\\"SLOW_OPS\\", ceph_daemon=~\\"osd.*\\"})\\n)\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Top Slow Ops",\n "transformations": [\n {\n "id": "merge",\n "options": {\n "reducers": [ ]\n }\n },\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time": true,\n "__name__": true,\n "cluster": true,\n "instance": true,\n "job": true,\n "type": true\n },\n "includeByName": { },\n "indexByName": { },\n "renameByName": { }\n }\n }\n ],\n "type": "table"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "OSD Overview",\n "uid": "lo02I1Aiz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/rbd-overview.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.4.2"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n },\n {\n "id": "prometheus",\n "name": "Prometheus",\n "type": "datasource",\n "version": "5.0.0"\n },\n {\n "id": "table",\n "name": "Table",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [\n {\n "asDropdown": true,\n "icon": "external link",\n "includeVars": true,\n "keepTime": true,\n "tags": [ ],\n "targetBlank": false,\n "title": "Browse Dashboards",\n "tooltip": "",\n "type": "dashboards",\n "url": ""\n }\n ],\n "panels": [\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "round(sum(rate(ceph_rbd_write_ops{cluster=~\\"$cluster|\\", }[$__rate_interval])))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Writes",\n "refId": "A"\n },\n {\n "expr": "round(sum(rate(ceph_rbd_read_ops{cluster=~\\"$cluster|\\", }[$__rate_interval])))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Reads",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "IOPS",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "Bps"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 8,\n "y": 0\n },\n "id": 3,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "round(sum(rate(ceph_rbd_write_bytes{cluster=~\\"$cluster|\\", }[$__rate_interval])))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Write",\n "refId": "A"\n },\n {\n "expr": "round(sum(rate(ceph_rbd_read_bytes{cluster=~\\"$cluster|\\", }[$__rate_interval])))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Read",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Throughput",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "Bps",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "aliasColors": { },\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "ns"\n }\n },\n "fill": 1,\n "fillGradient": 0,\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 16,\n "y": 0\n },\n "id": 4,\n "legend": {\n "alignAsTable": false,\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "rightSide": false,\n "show": true,\n "sideWidth": null,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [ ],\n "nullPointMode": "null as zero",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "repeat": null,\n "seriesOverrides": [ ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "round(\\n sum(rate(ceph_rbd_write_latency_sum{cluster=~\\"$cluster|\\", }[$__rate_interval])) /\\n sum(rate(ceph_rbd_write_latency_count{cluster=~\\"$cluster|\\", }[$__rate_interval]))\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Write",\n "refId": "A"\n },\n {\n "expr": "round(\\n sum(rate(ceph_rbd_read_latency_sum{cluster=~\\"$cluster|\\", }[$__rate_interval])) /\\n sum(rate(ceph_rbd_read_latency_count{cluster=~\\"$cluster|\\", }[$__rate_interval]))\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Read",\n "refId": "B"\n }\n ],\n "thresholds": [ ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Average Latency",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": [ ]\n },\n "yaxes": [\n {\n "format": "ns",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": 0,\n "show": true\n }\n ]\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "null",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "pool"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Pool"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "image"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Image"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "IOPS"\n },\n {\n "id": "unit",\n "value": "iops"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 0,\n "y": 7\n },\n "id": 5,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "10.4.0",\n "styles": [ ],\n "targets": [\n {\n "expr": "topk(10,\\n (\\n sort((\\n rate(ceph_rbd_write_ops{cluster=~\\"$cluster|\\", }[$__rate_interval]) +\\n on (image, pool, namespace) rate(ceph_rbd_read_ops{cluster=~\\"$cluster|\\", }[$__rate_interval])\\n ))\\n )\\n)\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Highest IOPS",\n "transformations": [\n {\n "id": "merge",\n "options": {\n "reducers": [ ]\n }\n }\n ],\n "type": "table"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "null",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "pool"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Pool"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "image"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Image"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Throughput"\n },\n {\n "id": "unit",\n "value": "Bps"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 8,\n "y": 7\n },\n "id": 6,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "10.4.0",\n "styles": [ ],\n "targets": [\n {\n "expr": "topk(10,\\n sort(\\n sum(\\n rate(ceph_rbd_read_bytes{cluster=~\\"$cluster|\\", }[$__rate_interval]) +\\n rate(ceph_rbd_write_bytes{cluster=~\\"$cluster|\\", }[$__rate_interval])\\n ) by (pool, image, namespace)\\n )\\n)\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Highest Throughput",\n "transformations": [\n {\n "id": "merge",\n "options": {\n "reducers": [ ]\n }\n }\n ],\n "type": "table"\n },\n {\n "columns": [ ],\n "datasource": "${datasource}",\n "description": "RBD per-image IO statistics are disabled by default.\\n\\nPlease refer to https://docs.ceph.com/en/latest/mgr/prometheus/#rbd-io-statistics for information about how to enable those optionally.",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "align": "null",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": true,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "pool"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Pool"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "image"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Image"\n },\n {\n "id": "unit",\n "value": "short"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "displayName",\n "value": "Latency"\n },\n {\n "id": "unit",\n "value": "ns"\n },\n {\n "id": "decimals",\n "value": 2\n },\n {\n "id": "custom.align",\n "value": null\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 7,\n "w": 8,\n "x": 16,\n "y": 7\n },\n "id": 7,\n "links": [ ],\n "options": {\n "footer": {\n "countRows": false,\n "enablePagination": false,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "frameIndex": 1,\n "showHeader": true\n },\n "pluginVersion": "10.4.0",\n "styles": [ ],\n "targets": [\n {\n "expr": "topk(10,\\n sum(\\n rate(ceph_rbd_write_latency_sum{cluster=~\\"$cluster|\\", }[$__rate_interval]) /\\n clamp_min(rate(ceph_rbd_write_latency_count{cluster=~\\"$cluster|\\", }[$__rate_interval]), 1) +\\n rate(ceph_rbd_read_latency_sum{cluster=~\\"$cluster|\\", }[$__rate_interval]) /\\n clamp_min(rate(ceph_rbd_read_latency_count{cluster=~\\"$cluster|\\", }[$__rate_interval]), 1)\\n ) by (pool, image, namespace)\\n)\\n",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Highest Latency",\n "transformations": [\n {\n "id": "merge",\n "options": {\n "reducers": [ ]\n }\n }\n ],\n "type": "table"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin",\n "overview"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "RBD Overview",\n "uid": "41FrpeUiz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/ceph-nvmeof.json': '{\n "__inputs": [ ],\n "__requires": [ ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "Ceph NVMe-oF gateways overview",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [\n {\n "asDropdown": true,\n "icon": "external link",\n "includeVars": true,\n "keepTime": true,\n "tags": [ ],\n "targetBlank": false,\n "title": "Browse Dashboards",\n "tooltip": "",\n "type": "dashboards",\n "url": ""\n }\n ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "Overview",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": null,\n "result": {\n "index": 1,\n "text": "0"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "dark-green",\n "value": null\n }\n ]\n },\n "unit": "none"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 0,\n "y": 1\n },\n "id": 3,\n "interval": "1m",\n "links": [ ],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "count(count by (group) (ceph_nvmeof_gateway_info))",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Gateway Groups",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "noValue": "0",\n "thresholds": {\n "mode": "",\n "steps": [\n {\n "color": "#808080",\n "value": null\n },\n {\n "color": "red",\n "value": 1.0003\n }\n ]\n },\n "unit": ""\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "Down"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "red",\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Total"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "#a7a38b",\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Available"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "green",\n "mode": "fixed"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 3,\n "w": 5,\n "x": 3,\n "y": 1\n },\n "id": 4,\n "interval": "1m",\n "links": [ ],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "count(ceph_cephadm_daemon_status{service_type=\\"nvmeof\\"})",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Total",\n "range": false,\n "refId": "A"\n },\n {\n "datasource": "$datasource",\n "expr": "count(ceph_cephadm_daemon_status{service_type=\\"nvmeof\\"}==1 or ceph_cephadm_daemon_status{service_type=\\"nvmeof\\"}==2)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Available",\n "range": false,\n "refId": "B"\n },\n {\n "datasource": "$datasource",\n "expr": "count(ceph_cephadm_daemon_status{service_type=\\"nvmeof\\"}==0 or ceph_cephadm_daemon_status{service_type=\\"nvmeof\\"}==-1 or ceph_cephadm_daemon_status{service_type=\\"nvmeof\\"} == -2)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Down",\n "range": false,\n "refId": "C"\n }\n ],\n "title": "Total Gateways",\n "transparent": false,\n "type": "stat"\n },\n {\n "datasource": "$datasource",\n "description": "Gateways in error states",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 5,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "normal"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 0,\n "mappings": [ ],\n "noValue": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "options": "number"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "fixedColor": "orange",\n "mode": "shades"\n }\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 7,\n "x": 8,\n "y": 1\n },\n "id": 5,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": false\n },\n "tooltip": {\n "hideZeros": true,\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "group by (daemon_name) (ceph_cephadm_daemon_status{service_type=\\"nvmeof\\"} == 0)",\n "format": "",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "stopped - {{ daemon_name }} ",\n "range": true,\n "refId": "A"\n },\n {\n "datasource": "$datasource",\n "expr": "group by (daemon_name) (ceph_cephadm_daemon_status{service_type=\\"nvmeof\\"} == -1)",\n "format": "",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "error - {{ daemon_name }}",\n "range": true,\n "refId": "B"\n },\n {\n "datasource": "$datasource",\n "expr": "group by (daemon_name) (ceph_cephadm_daemon_status{service_type=\\"nvmeof\\"} == -2)",\n "format": "",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "unknown_state - {{ daemon_name }}",\n "range": true,\n "refId": "C"\n }\n ],\n "title": "Unhealthy Gateway Trend",\n "type": "timeseries"\n },\n {\n "datasource": {\n "type": "datasource",\n "uid": "grafana"\n },\n "gridPos": {\n "h": 8,\n "w": 9,\n "x": 15,\n "y": 1\n },\n "id": 6,\n "limit": 10,\n "onlyAlertsOnDashboard": true,\n "options": {\n "alertInstanceLabelFilter": "",\n "alertName": "NVMEOF",\n "dashboardAlerts": false,\n "groupBy": [ ],\n "groupMode": "default",\n "maxItems": 20,\n "sortOrder": 3,\n "stateFilter": {\n "error": true,\n "firing": true,\n "noData": false,\n "normal": false,\n "pending": true\n },\n "viewMode": "list"\n },\n "show": "current",\n "sortOrder": 1,\n "stateFilter": [ ],\n "title": "Active Alerts",\n "type": "alertlist"\n },\n {\n "datasource": "$datasource",\n "description": "Gateways in each group",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n }\n },\n "mappings": [ ]\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 5,\n "w": 8,\n "x": 0,\n "y": 4\n },\n "id": 7,\n "options": {\n "displayLabels": [ ],\n "legend": {\n "calcs": [ ],\n "displayMode": "table",\n "placement": "right",\n "showLegend": true,\n "values": [\n "value"\n ]\n },\n "pieType": "pie",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "tooltip": {\n "maxHeight": 600,\n "mode": "single",\n "sort": "none"\n }\n },\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "count by (group) (ceph_nvmeof_gateway_info)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Gateways per group",\n "type": "piechart"\n },\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 9\n },\n "id": 8,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "NVMeoF Group Overview - $group",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": null,\n "result": {\n "index": 1,\n "text": "0"\n }\n },\n "type": "special"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "none"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 0,\n "y": 10\n },\n "id": 9,\n "interval": "1m",\n "links": [ ],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "count(ceph_nvmeof_gateway_info{group=~\'$group\'})",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Gateways",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": null,\n "result": {\n "index": 1,\n "text": "0"\n }\n },\n "type": "special"\n }\n ],\n "noValue": "0",\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "purple",\n "value": null\n }\n ]\n },\n "unit": "none"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 3,\n "y": 10\n },\n "id": 10,\n "interval": "1m",\n "links": [ ],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "count(count by(nqn) (ceph_nvmeof_subsystem_metadata{group=~\'$group\',instance=~\'$gateway\'}))",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Subsystems",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": null,\n "result": {\n "index": 1,\n "text": "0"\n }\n },\n "type": "special"\n }\n ],\n "noValue": "0",\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "blue",\n "value": null\n }\n ]\n },\n "unit": "none"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 6,\n "y": 10\n },\n "id": 11,\n "interval": "1m",\n "links": [ ],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "count(count by (bdev_name) (\\n ceph_nvmeof_bdev_metadata{instance=~\'$gateway\'} * on (instance) group_left(group) ceph_nvmeof_gateway_info{group=~\'$group\'}\\n))",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Namespaces",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "Average connected clients",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": null,\n "result": {\n "index": 1,\n "text": "0"\n }\n },\n "type": "special"\n }\n ],\n "noValue": "0",\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "yellow",\n "value": null\n }\n ]\n },\n "unit": "none"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 3,\n "x": 9,\n "y": 10\n },\n "id": 12,\n "interval": "1m",\n "links": [ ],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "avg by (nqn) (sum by(gw_name, nqn) (ceph_nvmeof_host_connection_state{instance=~\'$gateway\'} * on (instance) group_left(group) ceph_nvmeof_gateway_info{group=~\'$group\'}))",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Connected Clients",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "The sum of capacity from all namespaces defined to subsystems",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": null,\n "result": {\n "index": 1,\n "text": "0"\n }\n },\n "type": "special"\n }\n ],\n "noValue": "0",\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "light-blue",\n "value": null\n }\n ]\n },\n "unit": "bytes"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 5,\n "x": 12,\n "y": 10\n },\n "id": 13,\n "interval": "1m",\n "links": [ ],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(avg by (group) (sum by(group, instance) (ceph_nvmeof_bdev_capacity_bytes{instance=~\'$gateway\'} * on (instance) group_left(group) ceph_nvmeof_gateway_info{group=~\'$group\'})))",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Capacity Exported",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "WARNING if any subsystem is defined with open/no security",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [\n {\n "id": 0,\n "options": {\n "match": "null",\n "result": {\n "color": "dark-green",\n "index": 0,\n "text": "OK"\n }\n },\n "type": "special"\n },\n {\n "id": 1,\n "options": {\n "from": 1,\n "result": {\n "color": "dark-yellow",\n "index": 1,\n "text": "WARNING"\n },\n "to": 9999\n },\n "type": "range"\n },\n {\n "id": 2,\n "options": {\n "0": {\n "color": "dark-green",\n "index": 2,\n "text": "OK"\n }\n },\n "type": "value"\n }\n ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "none"\n }\n },\n "gridPos": {\n "h": 3,\n "w": 7,\n "x": 17,\n "y": 10\n },\n "id": 14,\n "interval": "1m",\n "links": [ ],\n "options": {\n "colorMode": "background_solid",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "count(ceph_nvmeof_subsystem_metadata{allow_any_host=\'yes\',group=~\'$group\', instance=~\'$gateway\'}) ",\n "format": "",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Subsystem Security",\n "transparent": false,\n "type": "stat"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 5,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 5,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "auto",\n "spanNulls": false,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "decimals": 0,\n "mappings": [ ],\n "noValue": "0",\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 8,\n "w": 9,\n "x": 0,\n "y": 13\n },\n "id": 15,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "desc"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "count by (group) (ceph_nvmeof_gateway_info{group=~\'$group\',instance=~\'$gateway\'})",\n "format": "",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": true,\n "refId": "A"\n }\n ],\n "title": "Active gateways count",\n "type": "timeseries"\n },\n {\n "datasource": "${datasource}",\n "description": "Shows the highest average CPU on a gateway within the gateway group",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "decimals": 2,\n "displayName": "${__field.instance}",\n "mappings": [ ],\n "max": 100,\n "min": 0,\n "noValue": "0",\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "percentunit"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byType",\n "unit": "number"\n },\n "properties": [\n {\n "id": "unit",\n "value": "decbytes"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 8,\n "x": 9,\n "y": 13\n },\n "id": 16,\n "options": {\n "displayMode": "lcd",\n "maxVizHeight": 50,\n "minVizHeight": 16,\n "minVizWidth": 8,\n "namePlacement": "top",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "/^Value$/",\n "limit": 5,\n "values": true\n },\n "showUnfilled": true,\n "sizing": "manual",\n "valueMode": "text"\n },\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "topk(5, avg by(instance) (rate(ceph_nvmeof_reactor_seconds_total{mode=\'busy\',instance=~\'$gateway\'}[$__rate_interval])))",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Busiest Gateway CPU",\n "type": "bargauge"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "All gateways",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n },\n "unit": "locale"\n }\n },\n "gridPos": {\n "h": 4,\n "w": 7,\n "x": 17,\n "y": 13\n },\n "id": 17,\n "interval": "1m",\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum(rate(ceph_nvmeof_bdev_reads_completed_total{instance=~\'$gateway\'}[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Read",\n "range": true,\n "refId": "A"\n },\n {\n "datasource": "$datasource",\n "expr": "sum(rate(ceph_nvmeof_bdev_writes_completed_total{instance=~\'$gateway\'}[$__rate_interval]))",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "Write",\n "range": true,\n "refId": "B"\n }\n ],\n "title": "IOPS - $gateway",\n "transparent": false,\n "type": "stat"\n },\n {\n "colors": null,\n "datasource": "$datasource",\n "description": "All gateways",\n "fieldConfig": {\n "defaults": {\n "decimals": 0,\n "links": [ ],\n "mappings": [ ],\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "semi-dark-blue",\n "value": null\n }\n ]\n },\n "unit": "binBps"\n }\n },\n "gridPos": {\n "h": 4,\n "w": 7,\n "x": 17,\n "y": 17\n },\n "id": 18,\n "interval": "1m",\n "links": [ ],\n "options": {\n "colorMode": "none",\n "graphMode": "area",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n },\n "textMode": "auto"\n },\n "pluginVersion": "9.1.3",\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "sum (rate(ceph_nvmeof_bdev_read_bytes_total{instance=~\'$gateway\'}[$__rate_interval]))",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "Read",\n "range": true,\n "refId": "A"\n },\n {\n "datasource": "$datasource",\n "expr": "sum (rate(ceph_nvmeof_bdev_written_bytes_total{instance=~\'$gateway\'}[$__rate_interval]))",\n "format": "time_series",\n "instant": false,\n "intervalFactor": 1,\n "legendFormat": "Write",\n "range": true,\n "refId": "B"\n }\n ],\n "title": "Throughput - $gateway",\n "transparent": false,\n "type": "stat"\n },\n {\n "columns": [ ],\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "thresholds"\n },\n "custom": {\n "align": "auto",\n "cellOptions": {\n "type": "auto"\n },\n "filterable": false,\n "inspect": false\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "red",\n "value": 80\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "GW Version"\n },\n "properties": [\n {\n "id": "custom.width",\n "value": 110\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Count"\n },\n "properties": [\n {\n "id": "custom.width",\n "value": 80\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Address"\n },\n "properties": [\n {\n "id": "custom.width",\n "value": 112\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Group Name"\n },\n "properties": [\n {\n "id": "custom.width",\n "value": 128\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Group Name"\n },\n "properties": [\n {\n "id": "custom.filterable",\n "value": true\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Reactors"\n },\n "properties": [\n {\n "id": "custom.width",\n "value": 70\n },\n {\n "id": "custom.align",\n "value": "left"\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 12,\n "w": 17,\n "x": 0,\n "y": 21\n },\n "id": 19,\n "links": [ ],\n "options": {\n "cellHeight": "sm",\n "footer": {\n "countRows": false,\n "enablePagination": true,\n "fields": "",\n "reducer": [\n "sum"\n ],\n "show": false\n },\n "showHeader": true,\n "sortBy": [ ]\n },\n "pluginVersion": "9.1.3",\n "styles": [ ],\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "(count by(instance) (ceph_nvmeof_reactor_seconds_total{mode=\'busy\'})) * on (instance) group_left(group,hostname,version,addr) ceph_nvmeof_gateway_info{group=~\'$group\'}",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n }\n ],\n "timeFrom": null,\n "timeShift": null,\n "title": "Gateway Information",\n "transformations": [\n {\n "id": "organize",\n "options": {\n "excludeByName": {\n "Time": true,\n "Value": false,\n "__name__": true,\n "instance": true,\n "job": true,\n "name": true,\n "port": true,\n "spdk_version": true\n },\n "includeByName": { },\n "indexByName": {\n "Time": 0,\n "Value": 11,\n "__name__": 3,\n "addr": 4,\n "group": 1,\n "hostname": 2,\n "instance": 5,\n "job": 6,\n "name": 7,\n "port": 8,\n "spdk_version": 9,\n "version": 10\n },\n "renameByName": {\n "Value": "Reactors",\n "addr": "Address",\n "group": "Group Name",\n "hostname": "Hostname",\n "job": "",\n "version": "GW Version"\n }\n }\n }\n ],\n "type": "table"\n },\n {\n "datasource": "${datasource}",\n "description": "Show the subsystems by the count of namespaces they present to the client",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "continuous-blues"\n },\n "displayName": "${__field.nqn}",\n "mappings": [ ],\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n },\n "unit": "none"\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "links",\n "value": [\n {\n "title": "",\n "url": "d/feeuv1dno43r4deed/ceph-nvme-of-gateways-performance?var-subsystem=${__data.fields.nqn}"\n }\n ]\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 8,\n "w": 7,\n "x": 17,\n "y": 21\n },\n "id": 20,\n "options": {\n "displayMode": "basic",\n "maxVizHeight": 50,\n "minVizHeight": 16,\n "minVizWidth": 8,\n "namePlacement": "top",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "/^Value$/",\n "limit": 5,\n "values": true\n },\n "showUnfilled": true,\n "sizing": "manual",\n "valueMode": "text"\n },\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "topk(5, avg by (nqn) (ceph_nvmeof_subsystem_namespace_count{instance=~\'$gateway\'} * on (instance) group_left(group) ceph_nvmeof_gateway_info{group=~\'$group\'}))",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "{{nqn}}",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Subsystems by Namespace",\n "type": "bargauge"\n },\n {\n "datasource": "${datasource}",\n "description": "Show the subsystems by the count of namespaces they present to the client",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "continuous-purples"\n },\n "displayName": "${__field.nqn}",\n "mappings": [ ],\n "min": 0,\n "noValue": "0",\n "thresholds": {\n "mode": "absolute",\n "steps": [\n {\n "color": "green"\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "Value"\n },\n "properties": [\n {\n "id": "links",\n "value": [\n {\n "title": "",\n "url": "d/feeuv1dno43r4deed/ceph-nvme-of-gateways-performance?var-subsystem=${__data.fields.nqn}"\n }\n ]\n }\n ]\n }\n ]\n },\n "gridPos": {\n "h": 9,\n "w": 7,\n "x": 17,\n "y": 29\n },\n "id": 21,\n "options": {\n "displayMode": "basic",\n "maxVizHeight": 50,\n "minVizHeight": 16,\n "minVizWidth": 8,\n "namePlacement": "top",\n "orientation": "horizontal",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "/^Value$/",\n "limit": 5,\n "values": true\n },\n "showUnfilled": true,\n "sizing": "manual",\n "valueMode": "text"\n },\n "targets": [\n {\n "datasource": "$datasource",\n "expr": "topk(5, avg by (nqn) (sum by(gw_name, nqn) (ceph_nvmeof_host_connection_state{instance=~\'$gateway\'} * on (instance) group_left(group) ceph_nvmeof_gateway_info{group=~\'$group\'}))) ",\n "format": "table",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "__auto",\n "range": false,\n "refId": "A"\n }\n ],\n "title": "Top 5 Subsystems with Most Connected Clients",\n "type": "bargauge"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": "39",\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": {\n "text": {\n "selected": false,\n "text": "37788254-f1d2-11ef-b13a-02000488170e",\n "value": "37788254-f1d2-11ef-b13a-02000488170e"\n },\n "value": {\n "selected": false,\n "text": "37788254-f1d2-11ef-b13a-02000488170e",\n "value": "37788254-f1d2-11ef-b13a-02000488170e"\n }\n },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": {\n "text": {\n "selected": false,\n "text": "All",\n "value": "$__all"\n },\n "value": {\n "selected": false,\n "text": "All",\n "value": "$__all"\n }\n },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": "Gateway Group",\n "multi": false,\n "name": "group",\n "options": [ ],\n "query": "label_values(ceph_nvmeof_gateway_info,group)",\n "refresh": 2,\n "regex": "",\n "sort": 0,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": {\n "text": {\n "selected": false,\n "text": "All",\n "value": "$__all"\n },\n "value": {\n "selected": false,\n "text": "All",\n "value": "$__all"\n }\n },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": "Gateway Hostname",\n "multi": false,\n "name": "gateway",\n "options": [ ],\n "query": "label_values(ceph_nvmeof_gateway_info{group=~\'$group\'},hostname)",\n "refresh": 2,\n "regex": "",\n "sort": 0,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-6h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "Ceph NVMe-oF Gateways - Overview",\n "uid": "feeuv1dno43r4ddjhjssdd",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/osd-device-details.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.3.2"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [\n {\n "asDropdown": true,\n "icon": "external link",\n "includeVars": true,\n "keepTime": true,\n "tags": [ ],\n "targetBlank": false,\n "title": "Browse Dashboards",\n "tooltip": "",\n "type": "dashboards",\n "url": ""\n }\n ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "OSD Performance",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "s"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 0,\n "y": 1\n },\n "id": 3,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "read",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "rate(ceph_osd_op_r_latency_sum{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster|\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_r_latency_count{cluster=~\\"$cluster|\\", }[$__rate_interval])\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "read",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_osd_op_w_latency_sum{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster|\\", }[$__rate_interval]) /\\n on (ceph_daemon) rate(ceph_osd_op_w_latency_count{cluster=~\\"$cluster|\\", }[$__rate_interval])\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "write",\n "refId": "B"\n }\n ],\n "title": "$osd Latency",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 6,\n "y": 1\n },\n "id": 4,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "Reads",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "rate(ceph_osd_op_r{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster|\\", }[$__rate_interval])",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Reads",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_osd_op_w{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster|\\", }[$__rate_interval])",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Writes",\n "refId": "B"\n }\n ],\n "title": "$osd R/W IOPS",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "bytes"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 12,\n "y": 1\n },\n "id": 5,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "Read Bytes",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "rate(ceph_osd_op_r_out_bytes{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster|\\", }[$__rate_interval])",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Read Bytes",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_osd_op_w_in_bytes{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster|\\", }[$__rate_interval])",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Write Bytes",\n "refId": "B"\n }\n ],\n "title": "$osd R/W Bytes",\n "type": "timeseries"\n },\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 10\n },\n "id": 6,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "Physical Device Performance",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "s"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 0,\n "y": 11\n },\n "id": 7,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*Reads/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "(\\n label_replace(\\n rate(node_disk_read_time_seconds_total[$__rate_interval]) /\\n rate(node_disk_reads_completed_total[$__rate_interval]),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n ) and on (instance, device) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster|\\", },\\n \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n )\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{instance}}/{{device}} Reads",\n "refId": "A"\n },\n {\n "expr": "(\\n label_replace(\\n rate(node_disk_write_time_seconds_total[$__rate_interval]) /\\n rate(node_disk_writes_completed_total[$__rate_interval]),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\") and on (instance, device)\\n label_replace(\\n label_replace(\\n ceph_disk_occupation_human{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster|\\", }, \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n )\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{instance}}/{{device}} Writes",\n "refId": "B"\n }\n ],\n "title": "Physical Device Latency for $osd",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 6,\n "y": 11\n },\n "id": 8,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*Reads/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "label_replace(\\n rate(node_disk_writes_completed_total[$__rate_interval]),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) and on (instance, device) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster|\\", },\\n \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}} on {{instance}} Writes",\n "refId": "A"\n },\n {\n "expr": "label_replace(\\n rate(node_disk_reads_completed_total[$__rate_interval]),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) and on (instance, device) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster|\\", },\\n \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}} on {{instance}} Reads",\n "refId": "B"\n }\n ],\n "title": "Physical Device R/W IOPS for $osd",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "Bps"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 12,\n "y": 11\n },\n "id": 9,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*Reads/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "label_replace(\\n rate(node_disk_read_bytes_total[$__rate_interval]), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) and on (instance, device) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster|\\", },\\n \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{instance}} {{device}} Reads",\n "refId": "A"\n },\n {\n "expr": "label_replace(\\n rate(node_disk_written_bytes_total[$__rate_interval]), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) and on (instance, device) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster|\\", },\\n \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{instance}} {{device}} Writes",\n "refId": "B"\n }\n ],\n "title": "Physical Device R/W Bytes for $osd",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "percentunit"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 18,\n "y": 11\n },\n "id": 10,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "label_replace(\\n rate(node_disk_io_time_seconds_total[$__rate_interval]),\\n \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n) and on (instance, device) label_replace(\\n label_replace(\\n ceph_disk_occupation_human{ceph_daemon=~\\"$osd\\", cluster=~\\"$cluster|\\", }, \\"device\\", \\"$1\\", \\"device\\", \\"/dev/(.*)\\"\\n ), \\"instance\\", \\"$1\\", \\"instance\\", \\"([^:.]*).*\\"\\n)\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{device}} on {{instance}}",\n "refId": "A"\n }\n ],\n "title": "Physical Device Util% for $osd",\n "type": "timeseries"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "OSD",\n "multi": false,\n "name": "osd",\n "options": [ ],\n "query": "label_values(ceph_osd_metadata{cluster=~\\"$cluster|\\", }, ceph_daemon)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-3h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "OSD device details",\n "uid": "CrAHE0iZz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/cephfs-overview.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.3.2"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [\n {\n "asDropdown": true,\n "icon": "external link",\n "includeVars": true,\n "keepTime": true,\n "tags": [ ],\n "targetBlank": false,\n "title": "Browse Dashboards",\n "tooltip": "",\n "type": "dashboards",\n "url": ""\n }\n ],\n "panels": [\n {\n "collapse": false,\n "collapsed": false,\n "gridPos": {\n "h": 1,\n "w": 24,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "panels": [ ],\n "repeat": null,\n "repeatIteration": null,\n "repeatRowId": null,\n "showTitle": true,\n "title": "MDS Performance",\n "titleSize": "h6",\n "type": "row"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Reads(-) / Writes (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "none"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 0,\n "y": 1\n },\n "id": 3,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "/.*Reads/",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "sum(rate(ceph_objecter_op_r{ceph_daemon=~\\"($mds_servers).*\\", cluster=~\\"$cluster\\", }[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Read Ops",\n "refId": "A"\n },\n {\n "expr": "sum(rate(ceph_objecter_op_w{ceph_daemon=~\\"($mds_servers).*\\", cluster=~\\"$cluster\\", }[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Write Ops",\n "refId": "B"\n }\n ],\n "title": "MDS Workload - $mds_servers",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Client Requests",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "never",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "min": 0,\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "none"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 12,\n "y": 1\n },\n "id": 4,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "ceph_mds_server_handle_client_request{ceph_daemon=~\\"($mds_servers).*\\", cluster=~\\"$cluster\\", }",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "{{ceph_daemon}}",\n "refId": "A"\n }\n ],\n "title": "Client Request Load - $mds_servers",\n "type": "timeseries"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": true,\n "label": "MDS Server",\n "multi": false,\n "name": "mds_servers",\n "options": [ ],\n "query": "label_values(ceph_mds_inodes{cluster=~\\"$cluster\\", }, ceph_daemon)",\n "refresh": 1,\n "regex": "",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "MDS Performance",\n "uid": "tbO9LAiZz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/pool-detail.json': '{\n "__inputs": [ ],\n "__requires": [\n {\n "id": "grafana",\n "name": "Grafana",\n "type": "grafana",\n "version": "5.3.2"\n },\n {\n "id": "graph",\n "name": "Graph",\n "type": "panel",\n "version": "5.0.0"\n },\n {\n "id": "singlestat",\n "name": "Singlestat",\n "type": "panel",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": [\n {\n "builtIn": 1,\n "datasource": "-- Grafana --",\n "enable": true,\n "hide": true,\n "iconColor": "rgba(0, 211, 255, 1)",\n "name": "Annotations & Alerts",\n "showIn": 0,\n "tags": [ ],\n "type": "dashboard"\n }\n ]\n },\n "description": "",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "links": [\n {\n "asDropdown": true,\n "icon": "external link",\n "includeVars": true,\n "keepTime": true,\n "tags": [ ],\n "targetBlank": false,\n "title": "Browse Dashboards",\n "tooltip": "",\n "type": "dashboards",\n "url": ""\n }\n ],\n "panels": [\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": true,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "format": "percentunit",\n "gauge": {\n "maxValue": 1,\n "minValue": 0,\n "show": true,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 7,\n "w": 7,\n "x": 0,\n "y": 0\n },\n "id": 2,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": true\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "(ceph_pool_stored{cluster=~\\"$cluster|\\", } / (ceph_pool_stored{cluster=~\\"$cluster|\\", } + ceph_pool_max_avail{cluster=~\\"$cluster|\\", })) *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{name=~\\"$pool_name\\", cluster=~\\"$cluster|\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": ".7,.8",\n "title": "Capacity used",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": 100,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "description": "Time till pool is full assuming the average fill rate of the last 6 hours",\n "format": "s",\n "gauge": {\n "maxValue": false,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 7,\n "w": 5,\n "x": 7,\n "y": 0\n },\n "id": 3,\n "interval": null,\n "links": [ ],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": ""\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "(ceph_pool_max_avail{cluster=~\\"$cluster|\\", } / deriv(ceph_pool_stored{cluster=~\\"$cluster|\\", }[6h])) *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{name=~\\"$pool_name\\", cluster=~\\"$cluster|\\", } > 0\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "",\n "refId": "A"\n }\n ],\n "thresholds": "current",\n "title": "Time till full",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": false\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Objects out(-) / in(+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "ops"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 12,\n "x": 12,\n "y": 0\n },\n "id": 4,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "deriv(ceph_pool_objects{cluster=~\\"$cluster|\\", }[1m]) *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{name=~\\"$pool_name\\", cluster=~\\"$cluster|\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Objects per second",\n "refId": "A"\n }\n ],\n "title": "$pool_name Object Ingress/Egress",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "iops"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 12,\n "x": 0,\n "y": 7\n },\n "id": 5,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "reads",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "rate(ceph_pool_rd{cluster=~\\"$cluster|\\", }[$__rate_interval]) *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{name=~\\"$pool_name\\", cluster=~\\"$cluster|\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "reads",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_pool_wr{cluster=~\\"$cluster|\\", }[$__rate_interval]) *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{name=~\\"$pool_name\\", cluster=~\\"$cluster|\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "writes",\n "refId": "B"\n }\n ],\n "title": "$pool_name Client IOPS",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Read (-) / Write (+)",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "Bps"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 12,\n "x": 12,\n "y": 7\n },\n "id": 6,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [\n {\n "alias": "reads",\n "transform": "negative-Y"\n }\n ],\n "targets": [\n {\n "expr": "rate(ceph_pool_rd_bytes{cluster=~\\"$cluster|\\", }[$__rate_interval]) +\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{name=~\\"$pool_name\\", cluster=~\\"$cluster|\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "reads",\n "refId": "A"\n },\n {\n "expr": "rate(ceph_pool_wr_bytes{cluster=~\\"$cluster|\\", }[$__rate_interval]) +\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{name=~\\"$pool_name\\", cluster=~\\"$cluster|\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "writes",\n "refId": "B"\n }\n ],\n "title": "$pool_name Client Throughput",\n "type": "timeseries"\n },\n {\n "datasource": "$datasource",\n "description": "",\n "fieldConfig": {\n "defaults": {\n "color": {\n "mode": "palette-classic"\n },\n "custom": {\n "axisCenteredZero": false,\n "axisColorMode": "text",\n "axisLabel": "Objects",\n "axisPlacement": "auto",\n "barAlignment": 0,\n "drawStyle": "line",\n "fillOpacity": 8,\n "gradientMode": "none",\n "hideFrom": {\n "legend": false,\n "tooltip": false,\n "viz": false\n },\n "lineInterpolation": "linear",\n "lineWidth": 1,\n "pointSize": 0,\n "scaleDistribution": {\n "type": "linear"\n },\n "showPoints": "",\n "spanNulls": true,\n "stacking": {\n "group": "A",\n "mode": "none"\n },\n "thresholdsStyle": {\n "mode": "off"\n }\n },\n "thresholds": {\n "mode": "absolute",\n "steps": [ ]\n },\n "unit": "short"\n },\n "overrides": [ ]\n },\n "gridPos": {\n "h": 7,\n "w": 12,\n "x": 0,\n "y": 14\n },\n "id": 7,\n "options": {\n "legend": {\n "calcs": [ ],\n "displayMode": "list",\n "placement": "bottom",\n "showLegend": true\n },\n "tooltip": {\n "mode": "multi",\n "sort": "none"\n }\n },\n "pluginVersion": "9.1.3",\n "seriesOverrides": [ ],\n "targets": [\n {\n "expr": "ceph_pool_objects{cluster=~\\"$cluster|\\", } *\\n on(pool_id) group_left(instance, name) ceph_pool_metadata{name=~\\"$pool_name\\", cluster=~\\"$cluster|\\", }\\n",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Number of Objects",\n "refId": "A"\n }\n ],\n "title": "$pool_name Objects",\n "type": "timeseries"\n }\n ],\n "refresh": "30s",\n "rows": [ ],\n "schemaVersion": 22,\n "style": "dark",\n "tags": [\n "ceph-mixin"\n ],\n "templating": {\n "list": [\n {\n "current": {\n "text": "default",\n "value": "default"\n },\n "hide": 0,\n "label": "Data Source",\n "name": "datasource",\n "options": [ ],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "cluster",\n "multi": false,\n "name": "cluster",\n "options": [ ],\n "query": "label_values(ceph_health_status, cluster)",\n "refresh": 1,\n "regex": "(.*)",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n },\n {\n "allValue": null,\n "current": { },\n "datasource": "$datasource",\n "hide": 0,\n "includeAll": false,\n "label": "Pool Name",\n "multi": false,\n "name": "pool_name",\n "options": [ ],\n "query": "label_values(ceph_pool_metadata{cluster=~\\"$cluster|\\", }, name)",\n "refresh": 1,\n "regex": "",\n "sort": 1,\n "tagValuesQuery": "",\n "tags": [ ],\n "tagsQuery": "",\n "type": "query",\n "useTags": false\n }\n ]\n },\n "time": {\n "from": "now-1h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "Ceph Pool Details",\n "uid": "-xyV8KCiz",\n "version": 0\n}\n', '/etc/grafana/provisioning/dashboards/ceph-cluster.json': '{\n "__requires": [\n {\n "type": "grafana",\n "id": "grafana",\n "name": "Grafana",\n "version": "5.0.0"\n },\n {\n "type": "panel",\n "id": "graph",\n "name": "Graph",\n "version": "5.0.0"\n },\n {\n "type": "panel",\n "id": "heatmap",\n "name": "Heatmap",\n "version": "5.0.0"\n },\n {\n "type": "panel",\n "id": "singlestat",\n "name": "Singlestat",\n "version": "5.0.0"\n }\n ],\n "annotations": {\n "list": []\n },\n "description": "Ceph cluster overview",\n "editable": false,\n "gnetId": null,\n "graphTooltip": 0,\n "hideControls": false,\n "id": null,\n "iteration": 1525415495309,\n "links": [\n {\n "asDropdown": true,\n "icon": "external link",\n "includeVars": true,\n "keepTime": true,\n "tags": [],\n "targetBlank": false,\n "title": "Browse Dashboards",\n "tooltip": "",\n "type": "dashboards",\n "url": ""\n }\n ],\n "panels": [\n {\n "cacheTimeout": null,\n "colorBackground": true,\n "colorValue": false,\n "colors": [\n "rgba(50, 128, 45, 0.9)",\n "rgba(237, 129, 40, 0.9)",\n "rgb(255, 0, 0)"\n ],\n "datasource": "$datasource",\n "editable": false,\n "error": false,\n "format": "none",\n "gauge": {\n "maxValue": 100,\n "minValue": 0,\n "show": false,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 0,\n "y": 0\n },\n "hideTimeOverride": true,\n "id": 21,\n "interval": "1m",\n "links": [],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "span": 2,\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": false\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "ceph_health_status{cluster=~\'$cluster|\'}",\n "format": "time_series",\n "instant": true,\n "interval": "$interval",\n "intervalFactor": 1,\n "refId": "A",\n "step": 60\n }\n ],\n "thresholds": "1,2",\n "timeFrom": null,\n "title": "Health Status",\n "transparent": false,\n "type": "singlestat",\n "valueFontSize": "50%",\n "valueMaps": [\n {\n "op": "=",\n "text": "OK",\n "value": "0"\n },\n {\n "op": "=",\n "text": "WARN",\n "value": "1"\n },\n {\n "op": "=",\n "text": "ERR",\n "value": "2"\n }\n ],\n "valueName": "current"\n },\n {\n "colorMode": "Panel",\n "colors": {\n "crit": "rgb(255, 0, 0)",\n "disable": "rgba(128, 128, 128, 0.9)",\n "ok": "rgba(50, 128, 45, 0.9)",\n "warn": "rgba(237, 129, 40, 0.9)"\n },\n "cornerRadius": 0,\n "datasource": "$datasource",\n "displayName": "",\n "flipCard": false,\n "flipTime": 5,\n "fontFormat": "Regular",\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 6,\n "y": 0\n },\n "id": 43,\n "isAutoScrollOnOverflow": false,\n "isGrayOnNoData": false,\n "isHideAlertsOnDisable": false,\n "isIgnoreOKColors": false,\n "links": [],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n }\n },\n "targets": [\n {\n "aggregation": "Last",\n "alias": "All",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ceph_osd_metadata{cluster=~\'$cluster|\'})",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "All",\n "refId": "A",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "In",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_osd_in{cluster=~\'$cluster|\'})",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "In",\n "refId": "B",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "Out",\n "decimals": 2,\n "displayAliasType": "Warning / Critical",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_osd_in{cluster=~\'$cluster|\'} == bool 0)",\n "format": "time_series",\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "Out",\n "refId": "C",\n "units": "none",\n "valueHandler": "Number Threshold",\n "warn": 1\n },\n {\n "aggregation": "Last",\n "alias": "Up",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_osd_up{cluster=~\'$cluster|\'})",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Up",\n "refId": "D",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "Down",\n "crit": 2,\n "decimals": 2,\n "displayAliasType": "Warning / Critical",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_osd_up{cluster=~\'$cluster|\'} == bool 0)",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Down",\n "refId": "E",\n "units": "none",\n "valueHandler": "Number Threshold",\n "warn": 1\n }\n ],\n "title": "OSDs",\n "type": "stat",\n "fieldConfig": {\n "defaults": {\n "mappings": [],\n "thresholds": {\n "mode": "percentage",\n "steps": [\n {\n "color": "green",\n "value": null\n }\n ]\n }\n },\n "overrides": [\n {\n "matcher": {\n "id": "byName",\n "options": "All"\n },\n "properties": [\n {\n "id": "color",\n "value": {\n "mode": "fixed"\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Out"\n },\n "properties": [\n {\n "id": "thresholds",\n "value": {\n "mode": "percentage",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "orange",\n "value": 0.1\n },\n {\n "value": 10,\n "color": "red"\n }\n ]\n }\n }\n ]\n },\n {\n "matcher": {\n "id": "byName",\n "options": "Down"\n },\n "properties": [\n {\n "id": "thresholds",\n "value": {\n "mode": "percentage",\n "steps": [\n {\n "color": "green",\n "value": null\n },\n {\n "color": "orange",\n "value": 0.1\n },\n {\n "value": 10,\n "color": "red"\n }\n ]\n }\n }\n ]\n }\n ]\n }\n },\n {\n "clusterName": "",\n "colorMode": "Panel",\n "colors": {\n "crit": "rgba(245, 54, 54, 0.9)",\n "disable": "rgba(128, 128, 128, 0.9)",\n "ok": "rgba(50, 128, 45, 0.9)",\n "warn": "rgba(237, 129, 40, 0.9)"\n },\n "cornerRadius": 1,\n "datasource": "$datasource",\n "displayName": "",\n "flipCard": false,\n "flipTime": 5,\n "fontFormat": "Regular",\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 12,\n "y": 0\n },\n "id": 41,\n "isAutoScrollOnOverflow": false,\n "isGrayOnNoData": false,\n "isHideAlertsOnDisable": false,\n "isIgnoreOKColors": false,\n "links": [],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n }\n },\n "targets": [\n {\n "aggregation": "Last",\n "alias": "In Quorum",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "sum(ceph_mon_quorum_status{cluster=~\'$cluster|\'})",\n "format": "time_series",\n "interval": "",\n "intervalFactor": 1,\n "legendFormat": "In Quorum",\n "refId": "A",\n "units": "none",\n "valueHandler": "Text Only"\n },\n {\n "aggregation": "Last",\n "alias": "Total",\n "crit": 1,\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ceph_mon_quorum_status{cluster=~\'$cluster|\'})",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Total",\n "refId": "B",\n "units": "none",\n "valueHandler": "Text Only",\n "warn": 2\n },\n {\n "aggregation": "Last",\n "alias": "MONs out of Quorum",\n "crit": 1.6,\n "decimals": 2,\n "displayAliasType": "Warning / Critical",\n "displayType": "Annotation",\n "displayValueWithAlias": "Never",\n "expr": "count(ceph_mon_quorum_status{cluster=~\'$cluster|\'}) - sum(ceph_mon_quorum_status{cluster=~\'$cluster|\'})",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "MONs out of Quorum",\n "refId": "C",\n "units": "none",\n "valueHandler": "Number Threshold",\n "warn": 1.1\n }\n ],\n "title": "Monitors",\n "type": "stat"\n },\n {\n "colorMode": "Panel",\n "colors": {\n "crit": "rgba(245, 54, 54, 0.9)",\n "disable": "rgba(128, 128, 128, 0.9)",\n "ok": "rgba(50, 128, 45, 0.9)",\n "warn": "rgba(237, 129, 40, 0.9)"\n },\n "cornerRadius": 1,\n "datasource": "$datasource",\n "displayName": "",\n "flipCard": false,\n "flipTime": 5,\n "fontFormat": "Regular",\n "gridPos": {\n "h": 3,\n "w": 6,\n "x": 18,\n "y": 0\n },\n "id": 68,\n "isAutoScrollOnOverflow": false,\n "isGrayOnNoData": false,\n "isHideAlertsOnDisable": false,\n "isIgnoreOKColors": false,\n "links": [],\n "options": {\n "colorMode": "background",\n "graphMode": "none",\n "justifyMode": "auto",\n "orientation": "auto",\n "reduceOptions": {\n "calcs": [\n "lastNotNull"\n ],\n "fields": "",\n "values": false\n }\n },\n "targets": [\n {\n "aggregation": "Last",\n "alias": "Active",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ceph_mgr_status{cluster=~\'$cluster|\'} == 1) or vector(0)",\n "format": "time_series",\n "intervalFactor": 1,\n "instant": true,\n "legendFormat": "Active",\n "refId": "A",\n "units": "none",\n "valueHandler": "Number Threshold"\n },\n {\n "aggregation": "Last",\n "alias": "Standby",\n "decimals": 2,\n "displayAliasType": "Always",\n "displayType": "Regular",\n "displayValueWithAlias": "When Alias Displayed",\n "expr": "count(ceph_mgr_status{cluster=~\'$cluster|\'} == 0) or vector(0)",\n "format": "time_series",\n "instant": true,\n "intervalFactor": 1,\n "legendFormat": "Standby",\n "refId": "B",\n "units": "none",\n "valueHandler": "Number Threshold"\n }\n ],\n "title": "MGRs",\n "type": "stat"\n },\n {\n "cacheTimeout": null,\n "colorBackground": false,\n "colorValue": false,\n "colors": [\n "#299c46",\n "rgba(237, 129, 40, 0.89)",\n "#d44a3a"\n ],\n "datasource": "$datasource",\n "decimals": 2,\n "format": "percentunit",\n "gauge": {\n "maxValue": 1,\n "minValue": 0,\n "show": true,\n "thresholdLabels": false,\n "thresholdMarkers": true\n },\n "gridPos": {\n "h": 6,\n "w": 6,\n "x": 0,\n "y": 6\n },\n "id": 47,\n "interval": null,\n "links": [],\n "mappingType": 1,\n "mappingTypes": [\n {\n "name": "value to text",\n "value": 1\n },\n {\n "name": "range to text",\n "value": 2\n }\n ],\n "maxDataPoints": 100,\n "nullPointMode": "connected",\n "nullText": null,\n "postfix": "",\n "postfixFontSize": "50%",\n "prefix": "",\n "prefixFontSize": "50%",\n "rangeMaps": [\n {\n "from": "null",\n "text": "N/A",\n "to": "null"\n }\n ],\n "sparkline": {\n "fillColor": "rgba(31, 118, 189, 0.18)",\n "full": false,\n "lineColor": "rgb(31, 120, 193)",\n "show": true\n },\n "tableColumn": "",\n "targets": [\n {\n "expr": "sum(ceph_osd_stat_bytes_used{cluster=~\'$cluster|\'})/sum(ceph_osd_stat_bytes{cluster=~\'$cluster|\'})",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Used",\n "refId": "A"\n }\n ],\n "thresholds": "0.7,0.8",\n "title": "Capacity used",\n "type": "singlestat",\n "valueFontSize": "80%",\n "valueMaps": [\n {\n "op": "=",\n "text": "N/A",\n "value": "null"\n }\n ],\n "valueName": "current"\n },\n {\n "aliasColors": {},\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "short"\n }\n },\n "fill": 0,\n "gridPos": {\n "h": 6,\n "w": 9,\n "x": 6,\n "y": 6\n },\n "id": 53,\n "legend": {\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "show": true,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [],\n "nullPointMode": "null",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "seriesOverrides": [\n {\n "alias": "Active",\n "color": "#508642",\n "fill": 1,\n "stack": "A"\n },\n {\n "alias": "Total",\n "color": "#f9e2d2"\n },\n {\n "alias": "Degraded",\n "color": "#eab839"\n },\n {\n "alias": "Undersized",\n "color": "#f9934e"\n },\n {\n "alias": "Inconsistent",\n "color": "#e24d42"\n },\n {\n "alias": "Down",\n "color": "#bf1b00"\n },\n {\n "alias": "Inactive",\n "color": "#bf1b00",\n "fill": 4,\n "linewidth": 0,\n "stack": "A"\n }\n ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(ceph_pg_total{cluster=~\'$cluster|\'})",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Total",\n "refId": "A"\n },\n {\n "expr": "sum(ceph_pg_active{cluster=~\'$cluster|\'})",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Active",\n "refId": "B"\n },\n {\n "expr": "sum(ceph_pg_total{cluster=~\'$cluster|\'} - ceph_pg_active{cluster=~\'$cluster|\'})",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Inactive",\n "refId": "G"\n },\n {\n "expr": "sum(ceph_pg_undersized{cluster=~\'$cluster|\'})",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Undersized",\n "refId": "F"\n },\n {\n "expr": "sum(ceph_pg_degraded{cluster=~\'$cluster|\'})",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Degraded",\n "refId": "C"\n },\n {\n "expr": "sum(ceph_pg_inconsistent{cluster=~\'$cluster|\'})",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Inconsistent",\n "refId": "D"\n },\n {\n "expr": "sum(ceph_pg_down{cluster=~\'$cluster|\'})",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Down",\n "refId": "E"\n }\n ],\n "thresholds": [],\n "timeFrom": null,\n "timeShift": null,\n "title": "PG States",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": []\n },\n "yaxes": [\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": "0",\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": null,\n "show": false\n }\n ]\n },\n {\n "aliasColors": {},\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "ms"\n }\n },\n "fill": 0,\n "gridPos": {\n "h": 6,\n "w": 9,\n "x": 15,\n "y": 6\n },\n "id": 66,\n "legend": {\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "show": false,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [],\n "nullPointMode": "null",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "seriesOverrides": [\n {\n "alias": "Avg Apply Latency",\n "color": "#7eb26d"\n }\n ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "quantile(0.95, ceph_osd_apply_latency_ms{cluster=~\'$cluster|\'})",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Apply Latency P_95",\n "refId": "A"\n },\n {\n "expr": "quantile(0.95, ceph_osd_commit_latency_ms{cluster=~\'$cluster|\'})",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Commit Latency P_95",\n "refId": "B"\n },\n {\n "expr": "avg(ceph_osd_apply_latency_ms{cluster=~\'$cluster|\'})",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Avg Apply Latency",\n "refId": "C"\n },\n {\n "expr": "avg(ceph_osd_commit_latency_ms{cluster=~\'$cluster|\'})",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Avg Commit Latency",\n "refId": "D"\n }\n ],\n "thresholds": [],\n "timeFrom": null,\n "timeShift": null,\n "title": "OSD Latencies",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": []\n },\n "yaxes": [\n {\n "format": "ms",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": null,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": null,\n "show": true\n }\n ]\n },\n {\n "aliasColors": {},\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "Bps"\n }\n },\n "fill": 1,\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 0,\n "y": 9\n },\n "id": 45,\n "legend": {\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "show": true,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [],\n "nullPointMode": "null",\n "percentage": false,\n "pointradius": 0.5,\n "points": false,\n "renderer": "flot",\n "seriesOverrides": [\n {\n "alias": "Reads",\n "transform": "negative-Y"\n }\n ],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(rate(ceph_osd_op_w_in_bytes{cluster=~\'$cluster|\'}[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Writes",\n "refId": "A"\n },\n {\n "expr": "sum(rate(ceph_osd_op_r_out_bytes{cluster=~\'$cluster|\'}[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Reads",\n "refId": "B"\n }\n ],\n "thresholds": [],\n "timeFrom": null,\n "timeShift": null,\n "title": "Cluster I/O",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": []\n },\n "yaxes": [\n {\n "format": "Bps",\n "label": "Read (-) / Write (+)",\n "logBase": 1,\n "max": null,\n "min": null,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": null,\n "show": false\n }\n ]\n },\n {\n "aliasColors": {},\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "Bps"\n }\n },\n "fill": 1,\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 12,\n "y": 9\n },\n "id": 62,\n "legend": {\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "show": false,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [],\n "nullPointMode": "null",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "seriesOverrides": [],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(deriv(ceph_pool_stored{cluster=~\'$cluster|\'}[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "refId": "A"\n }\n ],\n "thresholds": [],\n "timeFrom": null,\n "timeShift": null,\n "title": "In-/Egress",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": []\n },\n "yaxes": [\n {\n "format": "Bps",\n "label": " Egress (-) / Ingress (+)",\n "logBase": 1,\n "max": null,\n "min": null,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": null,\n "show": false\n }\n ]\n },\n {\n "cards": {\n "cardPadding": null,\n "cardRound": 1\n },\n "color": {\n "cardColor": "rgb(0, 254, 255)",\n "colorScale": "sqrt",\n "colorScheme": "interpolateBlues",\n "exponent": 0.5,\n "min": null,\n "mode": "spectrum"\n },\n "dataFormat": "timeseries",\n "datasource": "$datasource",\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 0,\n "y": 15\n },\n "heatmap": {},\n "highlightCards": true,\n "id": 55,\n "legend": {\n "show": true\n },\n "links": [],\n "span": 12,\n "targets": [\n {\n "expr": "ceph_osd_stat_bytes_used{cluster=~\'$cluster|\'} / ceph_osd_stat_bytes{cluster=~\'$cluster|\'}",\n "format": "time_series",\n "interval": "1m",\n "intervalFactor": 1,\n "legendFormat": "Util (%)",\n "refId": "A",\n "step": 60\n }\n ],\n "timeFrom": null,\n "title": "OSD Capacity Utilization",\n "tooltip": {\n "show": true,\n "showHistogram": false\n },\n "type": "heatmap",\n "xAxis": {\n "show": true\n },\n "xBucketNumber": null,\n "xBucketSize": "",\n "yAxis": {\n "decimals": 2,\n "format": "percentunit",\n "logBase": 1,\n "max": null,\n "min": null,\n "show": true,\n "splitFactor": null\n },\n "yBucketNumber": null,\n "yBucketSize": null\n },\n {\n "cards": {\n "cardPadding": null,\n "cardRound": 1\n },\n "color": {\n "cardColor": "#b4ff00",\n "colorScale": "sqrt",\n "colorScheme": "interpolateBlues",\n "exponent": 0.5,\n "mode": "spectrum"\n },\n "dataFormat": "timeseries",\n "datasource": "$datasource",\n "gridPos": {\n "h": 9,\n "w": 6,\n "x": 6,\n "y": 15\n },\n "heatmap": {},\n "highlightCards": true,\n "id": 59,\n "legend": {\n "show": true\n },\n "links": [],\n "targets": [\n {\n "expr": "ceph_osd_numpg{cluster=~\'$cluster|\'}",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "#PGs",\n "refId": "A"\n }\n ],\n "title": "PGs per OSD",\n "tooltip": {\n "show": true,\n "showHistogram": false\n },\n "type": "heatmap",\n "xAxis": {\n "show": true\n },\n "xBucketNumber": null,\n "xBucketSize": "",\n "yAxis": {\n "decimals": null,\n "format": "none",\n "logBase": 1,\n "max": null,\n "min": null,\n "show": true,\n "splitFactor": null\n },\n "yBucketNumber": null,\n "yBucketSize": null\n },\n {\n "aliasColors": {},\n "bars": false,\n "dashLength": 10,\n "dashes": false,\n "datasource": "$datasource",\n "fieldConfig": {\n "defaults": {\n "custom": {\n "fillOpacity": 8,\n "showPoints": "never"\n },\n "unit": "ops"\n }\n },\n "fill": 0,\n "gridPos": {\n "h": 9,\n "w": 12,\n "x": 12,\n "y": 15\n },\n "id": 64,\n "legend": {\n "avg": false,\n "current": false,\n "max": false,\n "min": false,\n "show": false,\n "total": false,\n "values": false\n },\n "lines": true,\n "linewidth": 1,\n "links": [],\n "nullPointMode": "null",\n "percentage": false,\n "pointradius": 5,\n "points": false,\n "renderer": "flot",\n "seriesOverrides": [],\n "spaceLength": 10,\n "stack": false,\n "steppedLine": false,\n "targets": [\n {\n "expr": "sum(rate(ceph_osd_recovery_ops{cluster=~\'$cluster|\'}[$__rate_interval]))",\n "format": "time_series",\n "intervalFactor": 1,\n "legendFormat": "Op/s",\n "refId": "A"\n }\n ],\n "thresholds": [],\n "timeFrom": null,\n "timeShift": null,\n "title": "Recovery Rate",\n "tooltip": {\n "shared": true,\n "sort": 0,\n "value_type": "individual"\n },\n "type": "timeseries",\n "xaxis": {\n "buckets": null,\n "mode": "time",\n "name": null,\n "show": true,\n "values": []\n },\n "yaxes": [\n {\n "format": "ops",\n "label": "Recovery Ops/s",\n "logBase": 1,\n "max": null,\n "min": null,\n "show": true\n },\n {\n "format": "short",\n "label": null,\n "logBase": 1,\n "max": null,\n "min": null,\n "show": true\n }\n ]\n }\n ],\n "refresh": "30s",\n "schemaVersion": 16,\n "style": "dark",\n "tags": [\n "ceph",\n "cluster"\n ],\n "templating": {\n "list": [\n {\n "hide": 0,\n "label": null,\n "name": "datasource",\n "options": [],\n "query": "prometheus",\n "refresh": 1,\n "regex": "",\n "type": "datasource"\n },\n {\n "datasource": {\n "type": "prometheus",\n "uid": "${datasource}"\n },\n "definition": "label_values(ceph_health_status, cluster)",\n "hide": 0,\n "includeAll": false,\n "label": "Cluster",\n "multi": false,\n "name": "cluster",\n "options": [],\n "query": {\n "query": "label_values(ceph_health_status, cluster)",\n "refId": "StandardVariableQuery"\n },\n "refresh": 1,\n "regex": "",\n "skipUrlSync": false,\n "sort": 0,\n "type": "query"\n },\n {\n "auto": true,\n "auto_count": 300,\n "auto_min": "1m",\n "current": {\n "text": "auto",\n "value": "$__auto_interval_interval"\n },\n "datasource": null,\n "hide": 0,\n "includeAll": false,\n "label": "Interval",\n "multi": false,\n "name": "interval",\n "options": [\n {\n "selected": true,\n "text": "auto",\n "value": "$__auto_interval_interval"\n },\n {\n "selected": false,\n "text": "1m",\n "value": "1m"\n },\n {\n "selected": false,\n "text": "10m",\n "value": "10m"\n },\n {\n "selected": false,\n "text": "30m",\n "value": "30m"\n },\n {\n "selected": false,\n "text": "1h",\n "value": "1h"\n },\n {\n "selected": false,\n "text": "6h",\n "value": "6h"\n },\n {\n "selected": false,\n "text": "12h",\n "value": "12h"\n },\n {\n "selected": false,\n "text": "1d",\n "value": "1d"\n },\n {\n "selected": false,\n "text": "7d",\n "value": "7d"\n },\n {\n "selected": false,\n "text": "14d",\n "value": "14d"\n },\n {\n "selected": false,\n "text": "30d",\n "value": "30d"\n }\n ],\n "query": "1m,10m,30m,1h,6h,12h,1d,7d,14d,30d",\n "refresh": 2,\n "type": "interval"\n }\n ]\n },\n "time": {\n "from": "now-6h",\n "to": "now"\n },\n "timepicker": {\n "refresh_intervals": [\n "5s",\n "10s",\n "30s",\n "1m",\n "5m",\n "15m",\n "30m",\n "1h",\n "2h",\n "1d"\n ],\n "time_options": [\n "5m",\n "15m",\n "1h",\n "6h",\n "12h",\n "24h",\n "2d",\n "7d",\n "30d"\n ]\n },\n "timezone": "",\n "title": "Ceph - Cluster",\n "uid": "edtb0oxdq",\n "version": 1\n}\n'}}} 2026-02-14 01:41:46,901 7f43ed435e00 DEBUG Determined image: 'quay.io/ceph/grafana:12.3.1' 2026-02-14 01:41:46,915 7f43ed435e00 INFO Redeploy daemon grafana.trial014 ... 2026-02-14 01:41:53,580 7f43ed435e00 DEBUG firewalld.service is not enabled 2026-02-14 01:41:53,580 7f43ed435e00 DEBUG Not possible to open ports <[3000]>. firewalld.service is not available 2026-02-14 01:41:53,723 7fba7c80de00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 01:41:54,752 7f8527c45e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:41:54,965 7eff9249ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'list-networks'] 2026-02-14 01:41:55,121 7f5eb1268e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 01:41:55,138 7f5eb1268e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 01:41:55,138 7f5eb1268e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 01:42:56,019 7fe0347c4e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:43:56,293 7fb6ecc42e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 01:43:56,310 7fb6ecc42e00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 01:43:56,310 7fb6ecc42e00 INFO systemctl is present 2026-02-14 01:43:56,310 7fb6ecc42e00 INFO lvcreate is present 2026-02-14 01:43:56,335 7fb6ecc42e00 INFO Unit chronyd.service is enabled and running 2026-02-14 01:43:56,335 7fb6ecc42e00 INFO Host looks OK 2026-02-14 01:43:56,467 7fe3f59dae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:44:56,728 7f08dc5cfe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:45:57,173 7fd32bfbbe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:46:57,448 7f70fa60fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:47:57,721 7f4d74a23e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:48:57,990 7ff6b2c2fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:49:58,262 7f73f0039e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:50:58,537 7f0a79279e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:51:58,811 7fb06006be00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 01:51:59,800 7ff942472e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:53:00,062 7fb036c64e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:54:00,334 7ff4a9e3fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 01:54:00,351 7ff4a9e3fe00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 01:54:00,352 7ff4a9e3fe00 INFO systemctl is present 2026-02-14 01:54:00,352 7ff4a9e3fe00 INFO lvcreate is present 2026-02-14 01:54:00,376 7ff4a9e3fe00 INFO Unit chronyd.service is enabled and running 2026-02-14 01:54:00,376 7ff4a9e3fe00 INFO Host looks OK 2026-02-14 01:54:00,508 7fd998e35e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:55:00,768 7f541cbcee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:56:01,213 7f04d61dae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:57:01,486 7f54f3004e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:58:01,762 7ffac1df0e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 01:59:02,035 7f2eaf6dbe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:00:02,304 7f60ebc8ee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:01:02,573 7f63e62d0e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:02:02,844 7fe8c60dae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 02:02:03,827 7f32d8e91e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:03:04,088 7f5c4ec36e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:04:04,361 7fc715abce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 02:04:04,377 7fc715abce00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 02:04:04,377 7fc715abce00 INFO systemctl is present 2026-02-14 02:04:04,377 7fc715abce00 INFO lvcreate is present 2026-02-14 02:04:04,400 7fc715abce00 INFO Unit chronyd.service is enabled and running 2026-02-14 02:04:04,400 7fc715abce00 INFO Host looks OK 2026-02-14 02:04:04,533 7fc416474e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:05:04,793 7fd08f451e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:06:05,230 7f53b6672e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:07:05,502 7f8ec8685e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:08:05,771 7f1ef61dfe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:09:06,043 7fe81165fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:10:06,315 7f42e2898e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:11:06,586 7f834e623e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:12:06,856 7f09cee68e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 02:12:07,829 7ff118bbee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:12:08,043 7f3e46cdce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'list-networks'] 2026-02-14 02:12:08,201 7fa715a64e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 02:12:08,217 7fa715a64e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 02:12:08,217 7fa715a64e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 02:13:08,932 7fda85078e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:14:09,211 7fe7baca0e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 02:14:09,229 7fe7baca0e00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 02:14:09,229 7fe7baca0e00 INFO systemctl is present 2026-02-14 02:14:09,229 7fe7baca0e00 INFO lvcreate is present 2026-02-14 02:14:09,253 7fe7baca0e00 INFO Unit chronyd.service is enabled and running 2026-02-14 02:14:09,253 7fe7baca0e00 INFO Host looks OK 2026-02-14 02:14:09,384 7fc8e085ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:15:09,639 7f99f0a6ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:16:10,085 7f1291fc4e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:17:10,357 7f07064aee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:18:10,633 7ff7c2203e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:19:10,905 7f3d02a26e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:20:11,177 7fde0e1cde00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:21:11,449 7fbe9284ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:22:11,719 7fc91503ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 02:22:12,694 7fcd2fbd7e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:23:12,954 7f5ae6604e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:24:13,228 7fd146241e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 02:24:13,244 7fd146241e00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 02:24:13,244 7fd146241e00 INFO systemctl is present 2026-02-14 02:24:13,244 7fd146241e00 INFO lvcreate is present 2026-02-14 02:24:13,269 7fd146241e00 INFO Unit chronyd.service is enabled and running 2026-02-14 02:24:13,270 7fd146241e00 INFO Host looks OK 2026-02-14 02:24:13,405 7f2505e26e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:25:13,663 7f57bc824e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:26:14,102 7f5fc2214e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:27:14,375 7f9f9bc3fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:28:14,646 7f6717ecee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:29:14,917 7fa6c01f8e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:30:15,187 7f3a5041be00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:31:15,459 7f2cf444ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:32:15,726 7f6e26c62e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 02:32:16,696 7fde40692e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:33:16,956 7f9839c9ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:34:17,231 7fcc4c5c5e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 02:34:17,248 7fcc4c5c5e00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 02:34:17,249 7fcc4c5c5e00 INFO systemctl is present 2026-02-14 02:34:17,249 7fcc4c5c5e00 INFO lvcreate is present 2026-02-14 02:34:17,273 7fcc4c5c5e00 INFO Unit chronyd.service is enabled and running 2026-02-14 02:34:17,273 7fcc4c5c5e00 INFO Host looks OK 2026-02-14 02:34:17,403 7f71d0e59e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:35:17,661 7f1670029e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:36:18,101 7fa93e60fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:37:18,372 7f6af447de00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:38:18,645 7f381062ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:39:18,916 7fb404264e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:40:19,190 7f02bbe96e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:41:19,463 7f28d05e1e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:42:19,732 7f154382ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 02:42:20,718 7f0508496e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:42:20,932 7f40629c8e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'list-networks'] 2026-02-14 02:42:21,087 7f7a2944ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 02:42:21,104 7f7a2944ce00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 02:42:21,104 7f7a2944ce00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 02:43:21,845 7f4b63c8ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:44:22,124 7fdd462b2e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 02:44:22,141 7fdd462b2e00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 02:44:22,141 7fdd462b2e00 INFO systemctl is present 2026-02-14 02:44:22,141 7fdd462b2e00 INFO lvcreate is present 2026-02-14 02:44:22,165 7fdd462b2e00 INFO Unit chronyd.service is enabled and running 2026-02-14 02:44:22,165 7fdd462b2e00 INFO Host looks OK 2026-02-14 02:44:22,296 7f717b64ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:45:22,556 7fe45fc56e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:46:22,997 7f8f8f45be00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:47:23,271 7fee38671e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:48:23,544 7fdf5200ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:49:23,814 7f63f5e49e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:50:24,089 7f03c4262e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:51:24,358 7f529d217e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:52:24,635 7f5ef91d4e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 02:52:25,623 7f5d35805e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:53:25,883 7fe44b662e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:54:26,156 7fa8b0035e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 02:54:26,172 7fa8b0035e00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 02:54:26,173 7fa8b0035e00 INFO systemctl is present 2026-02-14 02:54:26,173 7fa8b0035e00 INFO lvcreate is present 2026-02-14 02:54:26,197 7fa8b0035e00 INFO Unit chronyd.service is enabled and running 2026-02-14 02:54:26,197 7fa8b0035e00 INFO Host looks OK 2026-02-14 02:54:26,328 7fd114fc3e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:55:26,590 7fbc8f1dde00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:56:27,028 7f76fa461e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:57:27,294 7fb9458cde00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:58:27,565 7f05d644de00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 02:59:27,835 7f9402468e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:00:28,105 7f2506df3e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:01:28,374 7fce988dbe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:02:28,644 7f1e21ebde00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 03:02:29,608 7fd96a20be00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:03:29,870 7efe26a88e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:04:30,138 7f5ba1fe7e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 03:04:30,154 7f5ba1fe7e00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 03:04:30,154 7f5ba1fe7e00 INFO systemctl is present 2026-02-14 03:04:30,154 7f5ba1fe7e00 INFO lvcreate is present 2026-02-14 03:04:30,178 7f5ba1fe7e00 INFO Unit chronyd.service is enabled and running 2026-02-14 03:04:30,178 7f5ba1fe7e00 INFO Host looks OK 2026-02-14 03:04:30,311 7fc4fb053e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:05:30,572 7f4432bc9e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:06:31,013 7f6a76bdee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:07:31,280 7f8f9c485e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:08:31,550 7efdcf6c6e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:09:31,825 7f4965a73e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:10:32,097 7fb360489e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:11:32,368 7fc1c9493e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:12:32,641 7f0fd0057e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 03:12:33,612 7f0b51e5fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:12:33,831 7f9969fd2e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'list-networks'] 2026-02-14 03:12:33,986 7fb63343ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 03:12:34,002 7fb63343ae00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 03:12:34,003 7fb63343ae00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 03:13:34,712 7f997f41ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:14:34,987 7fb2ea06be00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 03:14:35,002 7fb2ea06be00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 03:14:35,002 7fb2ea06be00 INFO systemctl is present 2026-02-14 03:14:35,003 7fb2ea06be00 INFO lvcreate is present 2026-02-14 03:14:35,028 7fb2ea06be00 INFO Unit chronyd.service is enabled and running 2026-02-14 03:14:35,028 7fb2ea06be00 INFO Host looks OK 2026-02-14 03:14:35,162 7f3dd041ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:15:35,426 7ff5ba075e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:16:35,860 7f889dc5de00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:17:36,134 7f7a48ca0e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:18:36,408 7f78a0fc2e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:19:36,679 7f0203052e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:20:36,947 7f3d3b277e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:21:37,217 7f9942658e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:22:37,486 7fdd837e6e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 03:22:38,457 7f7d87bcce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:23:38,714 7f63b9e0be00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:24:38,988 7f4456a31e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 03:24:39,005 7f4456a31e00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 03:24:39,005 7f4456a31e00 INFO systemctl is present 2026-02-14 03:24:39,005 7f4456a31e00 INFO lvcreate is present 2026-02-14 03:24:39,028 7f4456a31e00 INFO Unit chronyd.service is enabled and running 2026-02-14 03:24:39,028 7f4456a31e00 INFO Host looks OK 2026-02-14 03:24:39,159 7f0a373bce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:25:39,413 7fa01d05ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:26:39,857 7f69f80b6e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:27:40,127 7f7543bfae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:28:40,392 7f29f1e73e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:29:40,667 7fdb31bd8e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:30:40,942 7f3ee5dcae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:31:41,217 7f14f1a1de00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:32:41,487 7f7fec64ee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 03:32:42,448 7f9cd4085e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:33:42,709 7efe8448ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:34:42,981 7fd846240e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 03:34:42,997 7fd846240e00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 03:34:42,997 7fd846240e00 INFO systemctl is present 2026-02-14 03:34:42,997 7fd846240e00 INFO lvcreate is present 2026-02-14 03:34:43,021 7fd846240e00 INFO Unit chronyd.service is enabled and running 2026-02-14 03:34:43,021 7fd846240e00 INFO Host looks OK 2026-02-14 03:34:43,153 7f612d3c6e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:35:43,411 7f7accc07e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:36:43,849 7ff1a87eae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:37:44,122 7f8deadbae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:38:44,389 7f6aa5a33e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:39:44,660 7f82e0c1ee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:40:44,931 7fe76a5efe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:41:45,200 7f9d9387ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:42:45,469 7f711684ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 03:42:46,465 7f0423cbde00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:42:46,680 7f06687d5e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'list-networks'] 2026-02-14 03:42:46,836 7fe8b4a26e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 03:42:46,852 7fe8b4a26e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 03:42:46,852 7fe8b4a26e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 03:43:47,605 7fa450439e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:44:47,880 7fe80bc04e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 03:44:47,896 7fe80bc04e00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 03:44:47,897 7fe80bc04e00 INFO systemctl is present 2026-02-14 03:44:47,897 7fe80bc04e00 INFO lvcreate is present 2026-02-14 03:44:47,921 7fe80bc04e00 INFO Unit chronyd.service is enabled and running 2026-02-14 03:44:47,921 7fe80bc04e00 INFO Host looks OK 2026-02-14 03:44:48,054 7ffb6c7dde00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:45:48,313 7f11f9260e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:46:48,760 7f1d89a90e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:47:49,036 7f374dd23e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:48:49,311 7f8b8dff2e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:49:49,583 7f1d3aeb8e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:50:49,852 7f7ff0e90e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:51:50,127 7f74889d9e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:52:50,398 7f072ec2ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 03:52:51,368 7fe5c6e52e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:53:51,627 7f2789e6ee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:54:51,901 7fe6553ede00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 03:54:51,918 7fe6553ede00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 03:54:51,918 7fe6553ede00 INFO systemctl is present 2026-02-14 03:54:51,918 7fe6553ede00 INFO lvcreate is present 2026-02-14 03:54:51,941 7fe6553ede00 INFO Unit chronyd.service is enabled and running 2026-02-14 03:54:51,941 7fe6553ede00 INFO Host looks OK 2026-02-14 03:54:52,073 7f6f7ea71e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:55:52,330 7f9d3f094e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:56:52,771 7f6a6461fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:57:53,043 7fac859c2e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:58:53,315 7f8e753bee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 03:59:53,586 7faceb018e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:00:53,857 7fb2d99f1e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:01:54,125 7f33f2235e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:02:54,399 7fd1e085be00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 04:02:55,381 7f4825c4be00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:03:55,642 7f2d583e5e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:04:55,915 7f34c289fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 04:04:55,931 7f34c289fe00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 04:04:55,932 7f34c289fe00 INFO systemctl is present 2026-02-14 04:04:55,932 7f34c289fe00 INFO lvcreate is present 2026-02-14 04:04:55,955 7f34c289fe00 INFO Unit chronyd.service is enabled and running 2026-02-14 04:04:55,955 7f34c289fe00 INFO Host looks OK 2026-02-14 04:04:56,085 7f07f8e3de00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:05:56,339 7f9adc456e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:06:56,786 7f3f41e71e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:07:57,057 7f8988beee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:08:57,329 7f6bb4de3e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:09:57,609 7f426bfc1e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:10:57,881 7feadcdc9e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:11:58,153 7f6cc6a67e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:12:58,430 7fdbdd44ee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 04:12:59,414 7f208c9eae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:12:59,625 7fc6d33eee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'list-networks'] 2026-02-14 04:12:59,780 7fa9e4a3ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 04:12:59,796 7fa9e4a3ce00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 04:12:59,796 7fa9e4a3ce00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 04:14:00,542 7f31a1e39e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:15:00,817 7ffbf108ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 04:15:00,833 7ffbf108ae00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 04:15:00,834 7ffbf108ae00 INFO systemctl is present 2026-02-14 04:15:00,834 7ffbf108ae00 INFO lvcreate is present 2026-02-14 04:15:00,857 7ffbf108ae00 INFO Unit chronyd.service is enabled and running 2026-02-14 04:15:00,857 7ffbf108ae00 INFO Host looks OK 2026-02-14 04:15:00,991 7f16947c9e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:16:01,246 7f74127c3e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:17:01,688 7ff478e74e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:18:01,962 7fde605cee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:19:02,234 7f4da4839e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:20:02,503 7f9aae1cee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:21:02,778 7f7a69078e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:22:03,052 7f23c4ab2e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:23:03,321 7fb1b99e1e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 04:23:04,297 7f54726a5e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:24:04,560 7ff7b2a7fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:25:04,833 7f6917a6fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 04:25:04,849 7f6917a6fe00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 04:25:04,849 7f6917a6fe00 INFO systemctl is present 2026-02-14 04:25:04,849 7f6917a6fe00 INFO lvcreate is present 2026-02-14 04:25:04,874 7f6917a6fe00 INFO Unit chronyd.service is enabled and running 2026-02-14 04:25:04,874 7f6917a6fe00 INFO Host looks OK 2026-02-14 04:25:05,004 7f750b422e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:26:05,262 7fa333e61e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:27:05,704 7f7683a3be00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:28:05,974 7fa798bd7e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:29:06,248 7faf50dc6e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:30:06,521 7f76709f8e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:31:06,791 7f1e4bfbbe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:32:07,062 7fe730420e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:33:07,334 7f0a9b401e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 04:33:08,327 7f1a8ce0ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:34:08,588 7f360ae02e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:35:08,862 7ff7622cce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 04:35:08,878 7ff7622cce00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 04:35:08,878 7ff7622cce00 INFO systemctl is present 2026-02-14 04:35:08,878 7ff7622cce00 INFO lvcreate is present 2026-02-14 04:35:08,902 7ff7622cce00 INFO Unit chronyd.service is enabled and running 2026-02-14 04:35:08,902 7ff7622cce00 INFO Host looks OK 2026-02-14 04:35:09,034 7f2dca06ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:36:09,296 7f2dcb22be00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:37:09,734 7f234049ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:38:10,008 7f49da1e5e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:39:10,278 7f5e7a923e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:40:10,549 7fb89accfe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:41:10,821 7ff413036e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:42:11,095 7f7ee4c76e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:43:11,367 7fc6bd24fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 04:43:12,393 7f50b19cee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:43:12,611 7f9391e5ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'list-networks'] 2026-02-14 04:43:12,765 7f924bec6e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 04:43:12,781 7f924bec6e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 04:43:12,781 7f924bec6e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 04:44:13,527 7fcff000ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:45:13,798 7f6154846e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 04:45:13,815 7f6154846e00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 04:45:13,815 7f6154846e00 INFO systemctl is present 2026-02-14 04:45:13,815 7f6154846e00 INFO lvcreate is present 2026-02-14 04:45:13,839 7f6154846e00 INFO Unit chronyd.service is enabled and running 2026-02-14 04:45:13,839 7f6154846e00 INFO Host looks OK 2026-02-14 04:45:13,969 7fd6897d9e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:46:14,232 7f63427cce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:47:14,680 7f205c05de00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:48:14,951 7f60d461de00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:49:15,227 7f35684a2e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:50:15,500 7f96fbbeae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:51:15,773 7ff15588ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:52:16,049 7f5f5b839e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:53:16,322 7f52fd8bae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 04:53:17,303 7f47d6ff1e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:54:17,561 7f2058dd1e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:55:17,834 7fdc17e5ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 04:55:17,851 7fdc17e5ce00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 04:55:17,851 7fdc17e5ce00 INFO systemctl is present 2026-02-14 04:55:17,851 7fdc17e5ce00 INFO lvcreate is present 2026-02-14 04:55:17,875 7fdc17e5ce00 INFO Unit chronyd.service is enabled and running 2026-02-14 04:55:17,876 7fdc17e5ce00 INFO Host looks OK 2026-02-14 04:55:18,008 7fdf19609e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:56:18,267 7f8b99855e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:57:18,707 7fd4eb005e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:58:18,979 7f80d77e1e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 04:59:19,250 7fd219006e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:00:19,524 7f3b35ebae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:01:19,797 7fa0eb5e1e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:02:20,068 7fc39ee3de00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:03:20,335 7f7ac6085e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 05:03:21,317 7fceb4e27e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:04:21,578 7f302341ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:05:21,851 7fb5dffd1e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 05:05:21,867 7fb5dffd1e00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 05:05:21,867 7fb5dffd1e00 INFO systemctl is present 2026-02-14 05:05:21,867 7fb5dffd1e00 INFO lvcreate is present 2026-02-14 05:05:21,893 7fb5dffd1e00 INFO Unit chronyd.service is enabled and running 2026-02-14 05:05:21,893 7fb5dffd1e00 INFO Host looks OK 2026-02-14 05:05:22,027 7f4874e0de00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:06:22,290 7fea0a892e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:07:22,726 7f3df5846e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:08:22,997 7fbcf83bee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:09:23,267 7f56da1dee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:10:23,538 7f42d6c82e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:11:23,810 7f1bdbe0fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:12:24,082 7f493c1e6e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:13:24,354 7fd470810e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 05:13:25,318 7f432e49fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:13:25,536 7f1201bd4e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'list-networks'] 2026-02-14 05:13:25,690 7ff3f6cbce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 05:13:25,706 7ff3f6cbce00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 05:13:25,706 7ff3f6cbce00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 05:14:26,418 7fbe8f8b9e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:15:26,690 7fc4ec3e6e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 05:15:26,706 7fc4ec3e6e00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 05:15:26,706 7fc4ec3e6e00 INFO systemctl is present 2026-02-14 05:15:26,706 7fc4ec3e6e00 INFO lvcreate is present 2026-02-14 05:15:26,731 7fc4ec3e6e00 INFO Unit chronyd.service is enabled and running 2026-02-14 05:15:26,731 7fc4ec3e6e00 INFO Host looks OK 2026-02-14 05:15:26,865 7f6abc284e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:16:27,125 7fa2f6441e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:17:27,567 7f0cf5032e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:18:27,840 7f93697d1e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:19:28,112 7ff6b0645e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:20:28,386 7f342ebffe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:21:28,654 7fbb2e221e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:22:28,929 7f31e14ade00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:23:29,206 7f7a327dce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 05:23:30,176 7f988f123e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:24:30,440 7f7fbb5f8e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:25:30,718 7f69fb43de00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 05:25:30,735 7f69fb43de00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 05:25:30,735 7f69fb43de00 INFO systemctl is present 2026-02-14 05:25:30,736 7f69fb43de00 INFO lvcreate is present 2026-02-14 05:25:30,760 7f69fb43de00 INFO Unit chronyd.service is enabled and running 2026-02-14 05:25:30,760 7f69fb43de00 INFO Host looks OK 2026-02-14 05:25:30,893 7f165127ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:26:31,149 7f669d4dce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:27:31,589 7fe381826e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:28:31,861 7f697ceade00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:29:32,134 7f6fbd5bde00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:30:32,412 7f5b9c7f1e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:31:32,680 7f36215f1e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:32:32,953 7f58f68bde00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:33:33,223 7fbaede6ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 05:33:34,196 7f5848656e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:34:34,462 7f04d73cae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:35:34,734 7fbf8aa69e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 05:35:34,750 7fbf8aa69e00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 05:35:34,750 7fbf8aa69e00 INFO systemctl is present 2026-02-14 05:35:34,750 7fbf8aa69e00 INFO lvcreate is present 2026-02-14 05:35:34,774 7fbf8aa69e00 INFO Unit chronyd.service is enabled and running 2026-02-14 05:35:34,774 7fbf8aa69e00 INFO Host looks OK 2026-02-14 05:35:34,905 7f172663be00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:36:35,163 7f9af5454e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:37:35,601 7f8df5e2ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:38:35,872 7f39c340fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:39:36,145 7ff70d4cee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:40:36,416 7fc33a43fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:41:36,687 7ff3c7c13e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:42:36,953 7f4b53812e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:43:37,228 7fac3b7dde00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 05:43:38,193 7f741cbf2e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:43:38,409 7f1f6004de00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'list-networks'] 2026-02-14 05:43:38,566 7fc5c3fffe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 05:43:38,582 7fc5c3fffe00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 05:43:38,583 7fc5c3fffe00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 05:44:39,327 7f7fdac94e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:45:39,602 7f3fabdfee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 05:45:39,619 7f3fabdfee00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 05:45:39,619 7f3fabdfee00 INFO systemctl is present 2026-02-14 05:45:39,619 7f3fabdfee00 INFO lvcreate is present 2026-02-14 05:45:39,643 7f3fabdfee00 INFO Unit chronyd.service is enabled and running 2026-02-14 05:45:39,643 7f3fabdfee00 INFO Host looks OK 2026-02-14 05:45:39,780 7fb8329dbe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:46:40,040 7f7f74a00e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:47:40,489 7f9995fc8e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:48:40,759 7fcf113dbe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:49:41,030 7fb2ece84e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:50:41,310 7f7b5825fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:51:41,581 7f49e16d6e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:52:41,855 7f1eb286ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:53:42,125 7fcbcea2fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 05:53:43,103 7f0d06fc4e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:54:43,366 7fb5aae48e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:55:43,643 7fbf26deee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 05:55:43,659 7fbf26deee00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 05:55:43,659 7fbf26deee00 INFO systemctl is present 2026-02-14 05:55:43,659 7fbf26deee00 INFO lvcreate is present 2026-02-14 05:55:43,683 7fbf26deee00 INFO Unit chronyd.service is enabled and running 2026-02-14 05:55:43,683 7fbf26deee00 INFO Host looks OK 2026-02-14 05:55:43,817 7f7979e1de00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:56:44,075 7f0dbb9bce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:57:44,515 7f9c68c51e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:58:44,790 7f4b3a600e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 05:59:45,065 7f39d2875e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:00:45,340 7f705abf0e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:01:45,612 7fda8068be00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:02:45,885 7fe64826ee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:03:46,156 7f6aa467fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 06:03:47,116 7f9ac26bbe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:04:47,377 7fb948e9fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:05:47,649 7f86e04a7e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 06:05:47,668 7f86e04a7e00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 06:05:47,668 7f86e04a7e00 INFO systemctl is present 2026-02-14 06:05:47,668 7f86e04a7e00 INFO lvcreate is present 2026-02-14 06:05:47,691 7f86e04a7e00 INFO Unit chronyd.service is enabled and running 2026-02-14 06:05:47,691 7f86e04a7e00 INFO Host looks OK 2026-02-14 06:05:47,824 7f8141c8de00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:06:48,083 7f9da07dee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:07:48,526 7fac3a0a5e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:08:48,796 7faca3847e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:09:49,066 7feeec27ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:10:49,340 7f1d54fc6e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:11:49,615 7f90c1065e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:12:49,885 7f93c65bce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:13:50,155 7f7c4e0a1e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 06:13:51,127 7f9b5485ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:13:51,341 7f71d683ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'list-networks'] 2026-02-14 06:13:51,495 7fce45fcce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 06:13:51,511 7fce45fcce00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 06:13:51,511 7fce45fcce00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 06:14:52,250 7f1ea0852e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:15:52,522 7f2e53241e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 06:15:52,538 7f2e53241e00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 06:15:52,539 7f2e53241e00 INFO systemctl is present 2026-02-14 06:15:52,539 7f2e53241e00 INFO lvcreate is present 2026-02-14 06:15:52,563 7f2e53241e00 INFO Unit chronyd.service is enabled and running 2026-02-14 06:15:52,563 7f2e53241e00 INFO Host looks OK 2026-02-14 06:15:52,699 7f803e04fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:16:52,956 7fb6dbaa4e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:17:53,402 7f315da43e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:18:53,672 7f6b743e3e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:19:53,944 7f5776871e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:20:54,220 7f423c27de00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:21:54,493 7f5a86e8ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:22:54,764 7f710907fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:23:55,040 7fa569c57e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 06:23:56,012 7fd62b672e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:24:56,273 7fa0c3e17e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:25:56,549 7febb83c2e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 06:25:56,566 7febb83c2e00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 06:25:56,566 7febb83c2e00 INFO systemctl is present 2026-02-14 06:25:56,566 7febb83c2e00 INFO lvcreate is present 2026-02-14 06:25:56,590 7febb83c2e00 INFO Unit chronyd.service is enabled and running 2026-02-14 06:25:56,591 7febb83c2e00 INFO Host looks OK 2026-02-14 06:25:56,728 7f1c35fe1e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:26:56,991 7fa1193f9e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:27:57,432 7f717c4a8e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:28:57,704 7f93783c5e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:29:57,970 7f6910818e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:30:58,251 7f085e3fee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:31:58,522 7f5f04c3ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:32:58,798 7ffb56073e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:33:59,067 7f31c61e3e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 06:34:00,039 7fc3d68cbe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:35:00,295 7f39f5e19e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:36:00,563 7f4972850e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 06:36:00,579 7f4972850e00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 06:36:00,579 7f4972850e00 INFO systemctl is present 2026-02-14 06:36:00,579 7f4972850e00 INFO lvcreate is present 2026-02-14 06:36:00,604 7f4972850e00 INFO Unit chronyd.service is enabled and running 2026-02-14 06:36:00,604 7f4972850e00 INFO Host looks OK 2026-02-14 06:36:00,735 7ff8ec060e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:37:00,990 7fbe2282fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:38:01,436 7f6b1b017e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:39:01,711 7f783d031e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:40:01,980 7f7e17452e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:41:02,254 7fd5f06dce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:42:02,526 7fe155c81e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:43:02,795 7f0e0bbc7e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:44:03,062 7f95ffe67e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 06:44:04,041 7fae3eabfe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:44:04,255 7fc927fc3e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'list-networks'] 2026-02-14 06:44:04,413 7f6f34dfee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 06:44:04,429 7f6f34dfee00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 06:44:04,429 7f6f34dfee00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 06:45:05,187 7fb4d9ccce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:46:05,459 7fc2ef015e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 06:46:05,476 7fc2ef015e00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 06:46:05,476 7fc2ef015e00 INFO systemctl is present 2026-02-14 06:46:05,476 7fc2ef015e00 INFO lvcreate is present 2026-02-14 06:46:05,500 7fc2ef015e00 INFO Unit chronyd.service is enabled and running 2026-02-14 06:46:05,501 7fc2ef015e00 INFO Host looks OK 2026-02-14 06:46:05,631 7f4c4ca6be00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:47:05,892 7fde0d260e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:48:06,332 7f73b9a34e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:49:06,608 7fcdd24ade00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:50:06,879 7ff1de087e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:51:07,153 7f9fef4ade00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:52:07,429 7f5d4eebce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:53:07,702 7ff26f265e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:54:07,972 7f049fa9ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 06:54:08,958 7f3217bcce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:55:09,219 7f70d43fbe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:56:09,490 7f87364b3e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 06:56:09,507 7f87364b3e00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 06:56:09,507 7f87364b3e00 INFO systemctl is present 2026-02-14 06:56:09,507 7f87364b3e00 INFO lvcreate is present 2026-02-14 06:56:09,531 7f87364b3e00 INFO Unit chronyd.service is enabled and running 2026-02-14 06:56:09,531 7f87364b3e00 INFO Host looks OK 2026-02-14 06:56:09,665 7f4db17c2e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:57:09,924 7fd6821f3e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:58:10,376 7ff540723e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 06:59:10,645 7feec603fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:00:10,917 7fa9b4e6ae00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:01:11,192 7fed9c856e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:02:11,461 7f1aade24e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:03:11,733 7f1fa6fcbe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:04:12,006 7f8f53081e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 07:04:12,975 7fcbb2832e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:05:13,232 7f9e3887fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:06:13,510 7f5585fc7e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 07:06:13,527 7f5585fc7e00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 07:06:13,527 7f5585fc7e00 INFO systemctl is present 2026-02-14 07:06:13,527 7f5585fc7e00 INFO lvcreate is present 2026-02-14 07:06:13,550 7f5585fc7e00 INFO Unit chronyd.service is enabled and running 2026-02-14 07:06:13,550 7f5585fc7e00 INFO Host looks OK 2026-02-14 07:06:13,684 7fb18f281e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:07:13,943 7f99e79d1e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:08:14,378 7f7658ac0e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:09:14,648 7ff3794ace00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:10:14,919 7f1aa2a09e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:11:15,190 7f140bc96e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:12:15,457 7efff84d2e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:13:15,729 7fc283dcbe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:14:15,998 7f6957800e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 07:14:16,972 7fb91d7d0e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:14:17,186 7fd4e7a0de00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'list-networks'] 2026-02-14 07:14:17,341 7f3048bd6e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ceph-volume', '--fsid', '29ce0a10-0945-11f1-b2be-d404e6e7d460', '--', 'inventory', '--format=json-pretty', '--filter-for-batch'] 2026-02-14 07:14:17,356 7f3048bd6e00 INFO Inferring config /var/lib/ceph/29ce0a10-0945-11f1-b2be-d404e6e7d460/mon.trial014/config 2026-02-14 07:14:17,357 7f3048bd6e00 DEBUG Using specified fsid: 29ce0a10-0945-11f1-b2be-d404e6e7d460 2026-02-14 07:15:18,099 7f2e980b8e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:16:18,373 7f65450ade00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 07:16:18,389 7f65450ade00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 07:16:18,390 7f65450ade00 INFO systemctl is present 2026-02-14 07:16:18,390 7f65450ade00 INFO lvcreate is present 2026-02-14 07:16:18,414 7f65450ade00 INFO Unit chronyd.service is enabled and running 2026-02-14 07:16:18,414 7f65450ade00 INFO Host looks OK 2026-02-14 07:16:18,551 7f0c87628e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:17:18,812 7f87239d8e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:18:19,252 7fdaf42d4e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:19:19,522 7f630c442e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:20:19,795 7f887c82ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:21:20,066 7f085bc0be00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:22:20,336 7f220f488e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:23:20,607 7f5063437e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:24:20,878 7f69a4876e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:abf03b71f4ac40c96279c787082e23ab98b3be9e820368fa0520e0d3b0aa54c9', '--timeout', '895', 'ls'] 2026-02-14 07:24:21,863 7f57c33cde00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:25:22,118 7f868ba3ce00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts'] 2026-02-14 07:26:22,394 7f27605d0e00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'check-host'] 2026-02-14 07:26:22,410 7f27605d0e00 INFO podman (/usr/bin/podman) version 5.6.0 is present 2026-02-14 07:26:22,410 7f27605d0e00 INFO systemctl is present 2026-02-14 07:26:22,410 7f27605d0e00 INFO lvcreate is present 2026-02-14 07:26:22,433 7f27605d0e00 INFO Unit chronyd.service is enabled and running 2026-02-14 07:26:22,433 7f27605d0e00 INFO Host looks OK 2026-02-14 07:26:22,565 7f934144ee00 DEBUG -------------------------------------------------------------------------------- cephadm ['--timeout', '895', 'gather-facts']