2023-01-25T05:29:13.664 INFO:root:teuthology version: 0.0.1.dev22+g8e1bba1 2023-01-25T05:29:13.666 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_8}} --verbose --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-01-24_22:22:09-rados-wip-yuri2-testing-2023-01-23-0928-distro-default-smithi/7136839 --name yuriw-2023-01-24_22:22:09-rados-wip-yuri2-testing-2023-01-23-0928-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-01-24_22:22:09-rados-wip-yuri2-testing-2023-01-23-0928-distro-default-smithi/7136839/orig.config.yaml 2023-01-25T05:29:13.703 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-01-25T05:29:13.778 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-01-24_22:22:09-rados-wip-yuri2-testing-2023-01-23-0928-distro-default-smithi/7136839 branch: wip-yuri2-testing-2023-01-23-0928 description: rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_8}} email: yweinstein@radhat.com first_in_suite: false job_id: '7136839' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-01-24_22:22:09-rados-wip-yuri2-testing-2023-01-23-0928-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri2-testing-2023-01-23-0928 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd debug verify cached snaps: true osd debug verify missing on start: true osd mclock override recovery settings: true osd op queue: debug_random osd op queue cut off: debug_random flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 61996d73ab738798e4ee0055c86457f5c459da4a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 61996d73ab738798e4ee0055c86457f5c459da4a selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri2-testing-2023-01-23-0928 sha1: 61996d73ab738798e4ee0055c86457f5c459da4a owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 - osd.4 - osd.5 - osd.6 - osd.7 - osd.8 - osd.9 - client.0 sha1: 61996d73ab738798e4ee0055c86457f5c459da4a sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri2-testing-2023-01-23-0928 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_61996d73ab738798e4ee0055c86457f5c459da4a/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 61996d73ab738798e4ee0055c86457f5c459da4a targets: smithi203.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC55Zz3HM1j0J7oxuTWSJhvf5bmHHheR3iKfuBOghJmm13VaucE0tUh59xKnzSpKUY8McQvELXBroAXrLBmaPNqu44vmbptfI0EYPgZYkDELfwxxwCE6w9GPqARlnNlG9qTLb/ZkCke4Scfdv2BS52z14nlgntP43O3Yprvqoi2FRD5x7GmS769OiEpWg4N2UYFiyPNk5aumG6wbJfW7MUg414oKllbBScP4H4JodAlMmLgj1KolkzGqq2BdcPCJk3qilnuyzde46cPKWrorCFCVlZ/TvG/5r+lCIazNSBBf43qY/k+oG+M9EHFGEBv722w3X1cpreV4k7Q7CMa1X7q3SZ0InX9GurHNMcFkxu+Z1M9kzHFBi8aPM6SLtyk5HMd9em2xwj67Gt3WdSJcysteKg9D2VaZMKDsAEcw76zsr/URN5fuXCeotNy25dOpALkVREQxel0KPMaoID17pgUOuxqlTcb7xyuCoODb7lfcn2h0kR3JBTHViTK0wQVaG8= tasks: - install: null - ceph: conf: osd: osd max object name len: 400 osd max object namespace len: 64 log-ignorelist: - but it is still running - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - workunit: clients: all: - rados/test_health_warnings.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8e1bba12e584b7ae912e92e94afe9c075ab51fa3 timestamp: 2023-01-24_22:22:09 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.17735 2023-01-25T05:29:13.779 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_61996d73ab738798e4ee0055c86457f5c459da4a/qa; will attempt to use it 2023-01-25T05:29:13.780 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_61996d73ab738798e4ee0055c86457f5c459da4a/qa/tasks 2023-01-25T05:29:13.781 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-01-25T05:29:13.783 INFO:teuthology.task.internal:Checking packages... 2023-01-25T05:29:13.818 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '61996d73ab738798e4ee0055c86457f5c459da4a' 2023-01-25T05:29:13.818 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-01-25T05:29:13.819 INFO:teuthology.packaging:ref: None 2023-01-25T05:29:13.819 INFO:teuthology.packaging:tag: None 2023-01-25T05:29:13.819 INFO:teuthology.packaging:branch: wip-yuri2-testing-2023-01-23-0928 2023-01-25T05:29:13.820 INFO:teuthology.packaging:sha1: 61996d73ab738798e4ee0055c86457f5c459da4a 2023-01-25T05:29:13.820 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri2-testing-2023-01-23-0928 2023-01-25T05:29:13.971 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-1963.g61996d73 2023-01-25T05:29:13.971 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-01-25T05:29:13.978 INFO:teuthology.task.internal:no buildpackages task found 2023-01-25T05:29:13.978 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-01-25T05:29:13.986 INFO:teuthology.task.internal:Saving configuration 2023-01-25T05:29:13.997 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-01-25T05:29:14.004 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-25T05:29:14.027 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-01-24_22:22:09-rados-wip-yuri2-testing-2023-01-23-0928-distro-default-smithi/7136839', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-01-25 05:23:56.853050', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNqk42dejgId1BENIkoWvoxzoeniG4sL7Sg72lundQIwQ64N5fTnQ4b7GYtLpfwsfJBc2ohpROgEPIApLmhq7uI='} 2023-01-25T05:29:14.028 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-01-25T05:29:14.035 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'osd.8', 'osd.9', 'client.0'] 2023-01-25T05:29:14.036 INFO:teuthology.run_tasks:Running task console_log... 2023-01-25T05:29:14.073 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff9ef8725e0>, signals=[15]) 2023-01-25T05:29:14.074 INFO:teuthology.run_tasks:Running task internal.connect... 2023-01-25T05:29:14.081 INFO:teuthology.task.internal:Opening connections... 2023-01-25T05:29:14.081 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi203.front.sepia.ceph.com 2023-01-25T05:29:14.083 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T05:29:14.155 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-01-25T05:29:14.162 DEBUG:teuthology.orchestra.run.smithi203:> uname -m 2023-01-25T05:29:14.181 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64 2023-01-25T05:29:14.182 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release 2023-01-25T05:29:14.243 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="CentOS Stream" 2023-01-25T05:29:14.244 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="8" 2023-01-25T05:29:14.248 INFO:teuthology.orchestra.run.smithi203.stdout:ID="centos" 2023-01-25T05:29:14.248 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE="rhel fedora" 2023-01-25T05:29:14.249 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="8" 2023-01-25T05:29:14.249 INFO:teuthology.orchestra.run.smithi203.stdout:PLATFORM_ID="platform:el8" 2023-01-25T05:29:14.249 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="CentOS Stream 8" 2023-01-25T05:29:14.249 INFO:teuthology.orchestra.run.smithi203.stdout:ANSI_COLOR="0;31" 2023-01-25T05:29:14.249 INFO:teuthology.orchestra.run.smithi203.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-01-25T05:29:14.250 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://centos.org/" 2023-01-25T05:29:14.250 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-01-25T05:29:14.250 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-01-25T05:29:14.250 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-01-25T05:29:14.251 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2023-01-25T05:29:14.269 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-01-25T05:29:14.279 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-01-25T05:29:14.285 INFO:teuthology.task.internal:Checking for old test directory... 2023-01-25T05:29:14.286 DEBUG:teuthology.orchestra.run.smithi203:> test '!' -e /home/ubuntu/cephtest 2023-01-25T05:29:14.303 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-01-25T05:29:14.309 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-01-25T05:29:14.310 DEBUG:teuthology.orchestra.run.smithi203:> test -z $(ls -A /var/lib/ceph) 2023-01-25T05:29:14.367 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-01-25T05:29:14.404 INFO:teuthology.run_tasks:Running task kernel... 2023-01-25T05:29:14.423 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-01-25T05:29:14.423 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-01-25T05:29:14.424 DEBUG:teuthology.orchestra.run.smithi203:> test -f /run/.containerenv -o -f /.dockerenv 2023-01-25T05:29:14.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-25T05:29:14.442 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2023-01-25T05:29:14.499 INFO:teuthology.orchestra.run.smithi203.stdout:4.18.0-408.el8.x86_64 2023-01-25T05:29:14.499 INFO:teuthology.task.kernel:Running kernel on smithi203: 4.18.0-408.el8.x86_64 2023-01-25T05:29:14.500 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum install -y kernel 2023-01-25T05:29:18.307 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-8 - AppStream 30 MB/s | 27 MB 00:00 2023-01-25T05:29:23.999 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-8 - BaseOS 39 MB/s | 27 MB 00:00 2023-01-25T05:29:28.256 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-8 - Extras 157 kB/s | 18 kB 00:00 2023-01-25T05:29:28.641 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-8 - PowerTools 22 MB/s | 5.6 MB 00:00 2023-01-25T05:29:29.857 INFO:teuthology.orchestra.run.smithi203.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2023-01-25T05:29:31.033 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 13 MB/s | 13 MB 00:01 2023-01-25T05:29:34.317 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 386 kB/s | 24 kB 00:00 2023-01-25T05:29:37.440 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-4.18.0-408.el8.x86_64 is already installed. 2023-01-25T05:29:37.440 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-4.18.0-305.3.1.el8.x86_64 is already installed. 2023-01-25T05:29:37.602 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:29:37.602 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:29:37.603 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2023-01-25T05:29:37.603 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:29:37.603 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2023-01-25T05:29:37.603 INFO:teuthology.orchestra.run.smithi203.stdout: kernel x86_64 4.18.0-448.el8 CentOS-BaseOS 9.2 M 2023-01-25T05:29:37.604 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-core x86_64 4.18.0-448.el8 CentOS-BaseOS 41 M 2023-01-25T05:29:37.604 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules x86_64 4.18.0-448.el8 CentOS-BaseOS 34 M 2023-01-25T05:29:37.604 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:29:37.604 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2023-01-25T05:29:37.605 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:29:37.605 INFO:teuthology.orchestra.run.smithi203.stdout:Install 3 Packages 2023-01-25T05:29:37.605 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:29:37.606 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 85 M 2023-01-25T05:29:37.606 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 94 M 2023-01-25T05:29:37.606 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2023-01-25T05:29:38.068 INFO:teuthology.orchestra.run.smithi203.stdout:(1/3): kernel-4.18.0-448.el8.x86_64.rpm 20 MB/s | 9.2 MB 00:00 2023-01-25T05:29:38.827 INFO:teuthology.orchestra.run.smithi203.stdout:(2/3): kernel-modules-4.18.0-448.el8.x86_64.rpm 28 MB/s | 34 MB 00:01 2023-01-25T05:29:39.252 INFO:teuthology.orchestra.run.smithi203.stdout:(3/3): kernel-core-4.18.0-448.el8.x86_64.rpm 25 MB/s | 41 MB 00:01 2023-01-25T05:29:39.255 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2023-01-25T05:29:39.255 INFO:teuthology.orchestra.run.smithi203.stdout:Total 51 MB/s | 85 MB 00:01 2023-01-25T05:29:39.893 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2023-01-25T05:29:40.296 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2023-01-25T05:29:40.297 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2023-01-25T05:29:43.458 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2023-01-25T05:29:43.486 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2023-01-25T05:29:46.201 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2023-01-25T05:29:46.846 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-core-4.18.0-448.el8.x86_64 1/3 2023-01-25T05:29:50.700 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-core-4.18.0-448.el8.x86_64 1/3 2023-01-25T05:29:51.153 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-modules-4.18.0-448.el8.x86_64 2/3 2023-01-25T05:29:54.474 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-4.18.0-448.el8.x86_64 2/3 2023-01-25T05:29:54.720 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-4.18.0-448.el8.x86_64 3/3 2023-01-25T05:30:16.396 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-core-4.18.0-448.el8.x86_64 3/3 2023-01-25T05:30:16.422 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-4.18.0-448.el8.x86_64 3/3 2023-01-25T05:30:16.850 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-4.18.0-448.el8.x86_64 3/3 2023-01-25T05:30:16.851 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-4.18.0-448.el8.x86_64 1/3 2023-01-25T05:30:16.851 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-core-4.18.0-448.el8.x86_64 2/3 2023-01-25T05:30:17.414 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-modules-4.18.0-448.el8.x86_64 3/3 2023-01-25T05:30:17.414 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:30:17.415 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2023-01-25T05:30:17.415 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-4.18.0-448.el8.x86_64 kernel-core-4.18.0-448.el8.x86_64 2023-01-25T05:30:17.415 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-4.18.0-448.el8.x86_64 2023-01-25T05:30:17.415 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:30:17.415 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:30:17.535 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel --last | head -n 1 2023-01-25T05:30:17.732 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-4.18.0-448.el8.x86_64 Wed 25 Jan 2023 05:29:54 AM UTC 2023-01-25T05:30:17.734 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-448.el8.x86_64 2023-01-25T05:30:17.734 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-408.el8.x86_64 Expected: 4.18.0-448.el8.x86_64 2023-01-25T05:30:17.734 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-01-25T05:30:17.734 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-01-25T05:30:17.735 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-01-25T05:30:17.735 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi203.front.sepia.ceph.com, path=None, version=distro) 2023-01-25T05:30:17.735 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel --last | head -n 1 2023-01-25T05:30:17.928 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-4.18.0-448.el8.x86_64 Wed 25 Jan 2023 05:29:54 AM UTC 2023-01-25T05:30:17.930 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-448.el8.x86_64 2023-01-25T05:30:17.930 DEBUG:teuthology.orchestra.run.smithi203:> sudo rpm -qi grub2-tools 2023-01-25T05:30:17.964 INFO:teuthology.orchestra.run.smithi203.stdout:Name : grub2-tools 2023-01-25T05:30:17.965 INFO:teuthology.orchestra.run.smithi203.stdout:Epoch : 1 2023-01-25T05:30:17.965 INFO:teuthology.orchestra.run.smithi203.stdout:Version : 2.02 2023-01-25T05:30:17.965 INFO:teuthology.orchestra.run.smithi203.stdout:Release : 129.el8 2023-01-25T05:30:17.966 INFO:teuthology.orchestra.run.smithi203.stdout:Architecture: x86_64 2023-01-25T05:30:17.966 INFO:teuthology.orchestra.run.smithi203.stdout:Install Date: Sun 28 Aug 2022 01:03:50 PM UTC 2023-01-25T05:30:17.966 INFO:teuthology.orchestra.run.smithi203.stdout:Group : System Environment/Base 2023-01-25T05:30:17.966 INFO:teuthology.orchestra.run.smithi203.stdout:Size : 9265586 2023-01-25T05:30:17.966 INFO:teuthology.orchestra.run.smithi203.stdout:License : GPLv3+ 2023-01-25T05:30:17.967 INFO:teuthology.orchestra.run.smithi203.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-01-25T05:30:17.967 INFO:teuthology.orchestra.run.smithi203.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-01-25T05:30:17.967 INFO:teuthology.orchestra.run.smithi203.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-01-25T05:30:17.967 INFO:teuthology.orchestra.run.smithi203.stdout:Build Host : kbuilder.bsys.centos.org 2023-01-25T05:30:17.967 INFO:teuthology.orchestra.run.smithi203.stdout:Relocations : (not relocatable) 2023-01-25T05:30:17.968 INFO:teuthology.orchestra.run.smithi203.stdout:Packager : CentOS BuildSystem 2023-01-25T05:30:17.968 INFO:teuthology.orchestra.run.smithi203.stdout:Vendor : CentOS 2023-01-25T05:30:17.968 INFO:teuthology.orchestra.run.smithi203.stdout:URL : http://www.gnu.org/software/grub/ 2023-01-25T05:30:17.968 INFO:teuthology.orchestra.run.smithi203.stdout:Summary : Support tools for GRUB. 2023-01-25T05:30:17.969 INFO:teuthology.orchestra.run.smithi203.stdout:Description : 2023-01-25T05:30:17.969 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:30:17.969 INFO:teuthology.orchestra.run.smithi203.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-01-25T05:30:17.969 INFO:teuthology.orchestra.run.smithi203.stdout:customizable bootloader with modular architecture. It supports a rich 2023-01-25T05:30:17.969 INFO:teuthology.orchestra.run.smithi203.stdout:variety of kernel formats, file systems, computer architectures and 2023-01-25T05:30:17.970 INFO:teuthology.orchestra.run.smithi203.stdout:hardware devices. 2023-01-25T05:30:17.970 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:30:17.970 INFO:teuthology.orchestra.run.smithi203.stdout:This subpackage provides tools for support of all platforms. 2023-01-25T05:30:17.971 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-01-25T05:30:17.971 INFO:teuthology.task.kernel:Updating grub on smithi203 to boot 4.18.0-448.el8.x86_64 2023-01-25T05:30:17.972 DEBUG:teuthology.orchestra.run.smithi203:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-01-25T05:30:19.606 INFO:teuthology.orchestra.run.smithi203.stderr:Generating grub configuration file ... 2023-01-25T05:30:21.722 INFO:teuthology.orchestra.run.smithi203.stderr:done 2023-01-25T05:30:21.723 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2023-01-25T05:30:21.746 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.9TsiIva1fK 2023-01-25T05:30:21.747 DEBUG:teuthology.orchestra.run.smithi203:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.9TsiIva1fK 2023-01-25T05:30:21.849 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.9TsiIva1fK 2023-01-25T05:30:21.950 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.9TsiIva1fK is 6KB 2023-01-25T05:30:21.998 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.9TsiIva1fK 2023-01-25T05:30:22.015 DEBUG:teuthology.orchestra.run.smithi203:> sudo /bin/ls /boot/loader/entries || true 2023-01-25T05:30:22.080 INFO:teuthology.orchestra.run.smithi203.stdout:2ff282740cf84d4fa02ff60ed95fbbff-0-rescue.conf 2023-01-25T05:30:22.081 INFO:teuthology.orchestra.run.smithi203.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-305.3.1.el8.x86_64.conf 2023-01-25T05:30:22.081 INFO:teuthology.orchestra.run.smithi203.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-408.el8.x86_64.conf 2023-01-25T05:30:22.081 INFO:teuthology.orchestra.run.smithi203.stdout:2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-448.el8.x86_64.conf 2023-01-25T05:30:22.082 DEBUG:teuthology.orchestra.run.smithi203:> sudo grub2-set-default 2ff282740cf84d4fa02ff60ed95fbbff-4.18.0-448.el8.x86_64 2023-01-25T05:30:22.238 DEBUG:teuthology.orchestra.run.smithi203:> sudo shutdown -r now 2023-01-25T05:30:52.268 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-01-25T05:30:52.269 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2023-01-25T05:30:52.269 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T05:31:10.824 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2023-01-25T05:31:40.852 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2023-01-25T05:31:40.853 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T05:31:59.212 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2023-01-25T05:32:29.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2023-01-25T05:32:29.241 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T05:32:29.511 DEBUG:teuthology.orchestra.run.smithi203:> true 2023-01-25T05:32:29.769 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2023-01-25T05:32:29.769 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-448.el8.x86_64"... 2023-01-25T05:32:29.770 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2023-01-25T05:32:29.789 INFO:teuthology.orchestra.run.smithi203.stdout:4.18.0-448.el8.x86_64 2023-01-25T05:32:29.789 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-448.el8.x86_64 vs 4.18.0-448.el8.x86_64 2023-01-25T05:32:29.789 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-01-25T05:32:29.790 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-01-25T05:32:30.791 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-01-25T05:32:30.792 DEBUG:teuthology.orchestra.run.smithi203:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-01-25T05:32:30.883 INFO:teuthology.orchestra.run.smithi203.stdout:ttyS1 2023-01-25T05:32:30.896 DEBUG:teuthology.parallel:result is None 2023-01-25T05:32:30.897 INFO:teuthology.run_tasks:Running task internal.base... 2023-01-25T05:32:30.905 INFO:teuthology.task.internal:Creating test directory... 2023-01-25T05:32:30.905 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-01-25T05:32:30.926 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-01-25T05:32:30.933 INFO:teuthology.run_tasks:Running task internal.archive... 2023-01-25T05:32:30.947 INFO:teuthology.task.internal:Creating archive directory... 2023-01-25T05:32:30.947 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-01-25T05:32:30.997 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-01-25T05:32:31.004 INFO:teuthology.task.internal:Enabling coredump saving... 2023-01-25T05:32:31.004 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-01-25T05:32:31.077 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T05:32:31.089 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-25T05:32:31.089 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-01-25T05:32:31.097 INFO:teuthology.task.internal:Configuring sudo... 2023-01-25T05:32:31.097 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-01-25T05:32:31.126 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-01-25T05:32:31.142 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-01-25T05:32:31.142 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-01-25T05:32:31.185 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-25T05:32:31.269 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-25T05:32:31.325 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-25T05:32:31.409 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-25T05:32:31.453 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:32:31.453 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-01-25T05:32:31.520 DEBUG:teuthology.orchestra.run.smithi203:> sudo service rsyslog restart 2023-01-25T05:32:31.632 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-01-25T05:32:31.939 INFO:teuthology.run_tasks:Running task internal.timer... 2023-01-25T05:32:31.948 INFO:teuthology.task.internal:Starting timer... 2023-01-25T05:32:31.948 INFO:teuthology.run_tasks:Running task pcp... 2023-01-25T05:32:31.960 INFO:teuthology.run_tasks:Running task selinux... 2023-01-25T05:32:31.974 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-01-25T05:32:31.975 DEBUG:teuthology.orchestra.run.smithi203:> sudo service auditd rotate 2023-01-25T05:32:32.052 INFO:teuthology.orchestra.run.smithi203.stdout:Rotating logs: 2023-01-25T05:32:32.054 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-01-25T05:32:32.054 DEBUG:teuthology.orchestra.run.smithi203:> /usr/sbin/getenforce 2023-01-25T05:32:32.099 INFO:teuthology.orchestra.run.smithi203.stdout:Permissive 2023-01-25T05:32:32.099 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi203.front.sepia.ceph.com': 'permissive'} 2023-01-25T05:32:32.100 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2023-01-25T05:32:32.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-25T05:32:32.128 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-01-25T05:32:32.129 DEBUG:teuthology.orchestra.run.smithi203:> sudo /usr/sbin/setenforce permissive 2023-01-25T05:32:32.197 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-01-25T05:32:32.213 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-01-25T05:32:32.214 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-01-25T05:32:32.234 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-01-25T05:32:32.236 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi203.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-01-25T05:38:54.162 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi203.front.sepia.ceph.com')] 2023-01-25T05:38:54.164 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2023-01-25T05:38:54.165 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-25T05:38:54.236 DEBUG:teuthology.orchestra.run.smithi203:> true 2023-01-25T05:38:54.302 INFO:teuthology.run_tasks:Running task clock... 2023-01-25T05:38:54.316 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-01-25T05:38:54.316 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-01-25T05:38:54.317 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-25T05:38:54.375 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-01-25T05:38:54.389 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-01-25T05:38:54.454 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: ntpd: command not found 2023-01-25T05:38:54.468 INFO:teuthology.orchestra.run.smithi203.stdout:506 Cannot talk to daemon 2023-01-25T05:38:54.480 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-01-25T05:38:54.494 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-01-25T05:38:54.560 INFO:teuthology.orchestra.run.smithi203.stderr:bash: ntpq: command not found 2023-01-25T05:38:54.563 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-01-25T05:38:54.564 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2023-01-25T05:38:54.564 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-25T05:38:54.564 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-25T05:38:54.564 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-25T05:38:54.565 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-01-25T05:38:54.565 INFO:teuthology.run_tasks:Running task install... 2023-01-25T05:38:54.576 DEBUG:teuthology.task.install:project ceph 2023-01-25T05:38:54.576 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '61996d73ab738798e4ee0055c86457f5c459da4a'}} 2023-01-25T05:38:54.577 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '61996d73ab738798e4ee0055c86457f5c459da4a'} 2023-01-25T05:38:54.577 INFO:teuthology.task.install:Using flavor: default 2023-01-25T05:38:54.584 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-01-25T05:38:54.585 INFO:teuthology.task.install:extra packages: [] 2023-01-25T05:38:54.585 DEBUG:teuthology.orchestra.run.smithi203:> sudo lsb_release -is 2023-01-25T05:38:54.642 INFO:teuthology.orchestra.run.smithi203.stdout:CentOSStream 2023-01-25T05:38:54.643 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-25T05:38:54.644 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '61996d73ab738798e4ee0055c86457f5c459da4a', 'tag': None, 'wait_for_package': False} 2023-01-25T05:38:54.644 DEBUG:teuthology.orchestra.run.smithi203:> sudo lsb_release -is 2023-01-25T05:38:54.720 INFO:teuthology.orchestra.run.smithi203.stdout:CentOSStream 2023-01-25T05:38:54.721 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-25T05:38:54.721 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=61996d73ab738798e4ee0055c86457f5c459da4a 2023-01-25T05:38:54.877 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/centos/8/flavors/default/ 2023-01-25T05:38:54.878 INFO:teuthology.task.install.rpm:Package version is 18.0.0-1963.g61996d73 2023-01-25T05:38:55.012 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri2-testing-2023-01-23-0928/61996d73ab738798e4ee0055c86457f5c459da4a/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-01-25T05:38:55.013 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:38:55.013 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-01-25T05:38:55.041 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-01-25T05:38:55.042 DEBUG:teuthology.orchestra.run.smithi203:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/61996d73ab738798e4ee0055c86457f5c459da4a/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-01-25T05:38:55.110 DEBUG:teuthology.orchestra.run.smithi203:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-01-25T05:38:55.187 DEBUG:teuthology.orchestra.run.smithi203:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-01-25T05:38:55.255 INFO:teuthology.orchestra.run.smithi203.stdout:check_obsoletes = 1 2023-01-25T05:38:55.257 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum clean all 2023-01-25T05:38:55.507 INFO:teuthology.orchestra.run.smithi203.stdout:61 files removed 2023-01-25T05:38:55.535 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-radosgw 2023-01-25T05:38:56.257 INFO:teuthology.orchestra.run.smithi203.stdout:ceph packages for x86_64 259 kB/s | 80 kB 00:00 2023-01-25T05:38:56.543 INFO:teuthology.orchestra.run.smithi203.stdout:ceph noarch packages 38 kB/s | 9.4 kB 00:00 2023-01-25T05:38:56.810 INFO:teuthology.orchestra.run.smithi203.stdout:ceph source packages 6.2 kB/s | 1.5 kB 00:00 2023-01-25T05:38:57.513 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-8 - AppStream 41 MB/s | 27 MB 00:00 2023-01-25T05:39:03.288 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-8 - BaseOS 33 MB/s | 27 MB 00:00 2023-01-25T05:39:08.128 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-8 - Extras 27 kB/s | 18 kB 00:00 2023-01-25T05:39:08.572 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-8 - PowerTools 18 MB/s | 5.6 MB 00:00 2023-01-25T05:39:09.821 INFO:teuthology.orchestra.run.smithi203.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-01-25T05:39:10.931 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 14 MB/s | 13 MB 00:00 2023-01-25T05:39:14.248 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 406 kB/s | 24 kB 00:00 2023-01-25T05:39:17.686 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:39:17.689 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:39:17.689 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2023-01-25T05:39:17.689 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:39:17.690 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2023-01-25T05:39:17.690 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw x86_64 2:18.0.0-1963.g61996d73.el8 ceph 13 M 2023-01-25T05:39:17.690 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrading: 2023-01-25T05:39:17.690 INFO:teuthology.orchestra.run.smithi203.stdout: librados2 x86_64 2:18.0.0-1963.g61996d73.el8 ceph 3.6 M 2023-01-25T05:39:17.690 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1 x86_64 2:18.0.0-1963.g61996d73.el8 ceph 3.6 M 2023-01-25T05:39:17.691 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2023-01-25T05:39:17.691 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base x86_64 2:18.0.0-1963.g61996d73.el8 ceph 5.2 M 2023-01-25T05:39:17.691 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common x86_64 2:18.0.0-1963.g61996d73.el8 ceph 24 M 2023-01-25T05:39:17.691 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux x86_64 2:18.0.0-1963.g61996d73.el8 ceph 24 k 2023-01-25T05:39:17.691 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-01-25T05:39:17.691 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2 x86_64 2:18.0.0-1963.g61996d73.el8 ceph 767 k 2023-01-25T05:39:17.692 INFO:teuthology.orchestra.run.smithi203.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-01-25T05:39:17.692 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-BaseOS 47 k 2023-01-25T05:39:17.692 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1 x86_64 2:18.0.0-1963.g61996d73.el8 ceph 474 k 2023-01-25T05:39:17.692 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka x86_64 0.11.4-3.el8 CentOS-AppStream 354 k 2023-01-25T05:39:17.692 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2 x86_64 2:18.0.0-1963.g61996d73.el8 ceph 6.5 M 2023-01-25T05:39:17.693 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-01-25T05:39:17.693 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2023-01-25T05:39:17.693 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse 2023-01-25T05:39:17.693 INFO:teuthology.orchestra.run.smithi203.stdout: x86_64 2:18.0.0-1963.g61996d73.el8 ceph 45 k 2023-01-25T05:39:17.693 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common x86_64 2:18.0.0-1963.g61996d73.el8 ceph 115 k 2023-01-25T05:39:17.694 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs x86_64 2:18.0.0-1963.g61996d73.el8 ceph 190 k 2023-01-25T05:39:17.694 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2023-01-25T05:39:17.694 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados x86_64 2:18.0.0-1963.g61996d73.el8 ceph 371 k 2023-01-25T05:39:17.694 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd x86_64 2:18.0.0-1963.g61996d73.el8 ceph 365 k 2023-01-25T05:39:17.694 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw x86_64 2:18.0.0-1963.g61996d73.el8 ceph 107 k 2023-01-25T05:39:17.695 INFO:teuthology.orchestra.run.smithi203.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-01-25T05:39:17.695 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:39:17.695 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2023-01-25T05:39:17.695 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:39:17.695 INFO:teuthology.orchestra.run.smithi203.stdout:Install 21 Packages 2023-01-25T05:39:17.695 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrade 2 Packages 2023-01-25T05:39:17.696 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:39:17.696 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 62 M 2023-01-25T05:39:17.697 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2023-01-25T05:39:18.465 INFO:teuthology.orchestra.run.smithi203.stdout:(1/23): ceph-base-18.0.0-1963.g61996d73.el8.x86 6.7 MB/s | 5.2 MB 00:00 2023-01-25T05:39:18.516 INFO:teuthology.orchestra.run.smithi203.stdout:(2/23): ceph-selinux-18.0.0-1963.g61996d73.el8. 487 kB/s | 24 kB 00:00 2023-01-25T05:39:18.641 INFO:teuthology.orchestra.run.smithi203.stdout:(3/23): libcephfs2-18.0.0-1963.g61996d73.el8.x8 6.0 MB/s | 767 kB 00:00 2023-01-25T05:39:18.766 INFO:teuthology.orchestra.run.smithi203.stdout:(4/23): libradosstriper1-18.0.0-1963.g61996d73. 3.7 MB/s | 474 kB 00:00 2023-01-25T05:39:19.250 INFO:teuthology.orchestra.run.smithi203.stdout:(5/23): ceph-radosgw-18.0.0-1963.g61996d73.el8. 8.4 MB/s | 13 MB 00:01 2023-01-25T05:39:19.371 INFO:teuthology.orchestra.run.smithi203.stdout:(6/23): python3-ceph-argparse-18.0.0-1963.g6199 371 kB/s | 45 kB 00:00 2023-01-25T05:39:19.467 INFO:teuthology.orchestra.run.smithi203.stdout:(7/23): python3-ceph-common-18.0.0-1963.g61996d 1.2 MB/s | 115 kB 00:00 2023-01-25T05:39:19.584 INFO:teuthology.orchestra.run.smithi203.stdout:(8/23): librgw2-18.0.0-1963.g61996d73.el8.x86_6 7.9 MB/s | 6.5 MB 00:00 2023-01-25T05:39:19.610 INFO:teuthology.orchestra.run.smithi203.stdout:(9/23): python3-cephfs-18.0.0-1963.g61996d73.el 1.3 MB/s | 190 kB 00:00 2023-01-25T05:39:19.660 INFO:teuthology.orchestra.run.smithi203.stdout:(10/23): python3-rados-18.0.0-1963.g61996d73.el 4.8 MB/s | 371 kB 00:00 2023-01-25T05:39:19.694 INFO:teuthology.orchestra.run.smithi203.stdout:(11/23): python3-rbd-18.0.0-1963.g61996d73.el8. 4.3 MB/s | 365 kB 00:00 2023-01-25T05:39:19.719 INFO:teuthology.orchestra.run.smithi203.stdout:(12/23): python3-rgw-18.0.0-1963.g61996d73.el8. 1.8 MB/s | 107 kB 00:00 2023-01-25T05:39:19.953 INFO:teuthology.orchestra.run.smithi203.stdout:(13/23): ceph-common-18.0.0-1963.g61996d73.el8. 11 MB/s | 24 MB 00:02 2023-01-25T05:39:20.028 INFO:teuthology.orchestra.run.smithi203.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 660 kB/s | 44 kB 00:00 2023-01-25T05:39:20.053 INFO:teuthology.orchestra.run.smithi203.stdout:(15/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 777 kB/s | 259 kB 00:00 2023-01-25T05:39:20.079 INFO:teuthology.orchestra.run.smithi203.stdout:(16/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 930 kB/s | 47 kB 00:00 2023-01-25T05:39:20.104 INFO:teuthology.orchestra.run.smithi203.stdout:(17/23): librdkafka-0.11.4-3.el8.x86_64.rpm 863 kB/s | 354 kB 00:00 2023-01-25T05:39:20.205 INFO:teuthology.orchestra.run.smithi203.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 470 kB/s | 59 kB 00:00 2023-01-25T05:39:20.239 INFO:teuthology.orchestra.run.smithi203.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 561 kB/s | 75 kB 00:00 2023-01-25T05:39:20.288 INFO:teuthology.orchestra.run.smithi203.stdout:(20/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2023-01-25T05:39:20.456 INFO:teuthology.orchestra.run.smithi203.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 6.9 MB/s | 1.7 MB 00:00 2023-01-25T05:39:20.548 INFO:teuthology.orchestra.run.smithi203.stdout:(22/23): librados2-18.0.0-1963.g61996d73.el8.x8 12 MB/s | 3.6 MB 00:00 2023-01-25T05:39:20.673 INFO:teuthology.orchestra.run.smithi203.stdout:(23/23): librbd1-18.0.0-1963.g61996d73.el8.x86_ 9.3 MB/s | 3.6 MB 00:00 2023-01-25T05:39:20.676 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2023-01-25T05:39:20.676 INFO:teuthology.orchestra.run.smithi203.stdout:Total 21 MB/s | 62 MB 00:02 2023-01-25T05:39:20.713 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2023-01-25T05:39:20.758 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2023-01-25T05:39:20.758 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2023-01-25T05:39:21.378 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2023-01-25T05:39:21.381 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2023-01-25T05:39:22.606 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2023-01-25T05:39:22.780 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-01-25T05:39:23.028 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/25 2023-01-25T05:39:23.194 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-01-25T05:39:23.596 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/25 2023-01-25T05:39:23.827 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librados2-2:18.0.0-1963.g61996d73.el8.x86_64 4/25 2023-01-25T05:39:23.921 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:18.0.0-1963.g61996d73.el8.x86_64 4/25 2023-01-25T05:39:24.218 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rados-2:18.0.0-1963.g61996d73.el8.x86_64 5/25 2023-01-25T05:39:24.394 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs2-2:18.0.0-1963.g61996d73.el8.x86_64 6/25 2023-01-25T05:39:24.774 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs2-2:18.0.0-1963.g61996d73.el8.x86_64 6/25 2023-01-25T05:39:24.961 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librbd1-2:18.0.0-1963.g61996d73.el8.x86_64 7/25 2023-01-25T05:39:25.029 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:18.0.0-1963.g61996d73.el8.x86_64 7/25 2023-01-25T05:39:25.250 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2023-01-25T05:39:25.394 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-01-25T05:39:25.465 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-01-25T05:39:25.596 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-argparse-2:18.0.0-1963.g61996d73.el8. 10/25 2023-01-25T05:39:25.812 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cephfs-2:18.0.0-1963.g61996d73.el8.x86_64 11/25 2023-01-25T05:39:26.061 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rbd-2:18.0.0-1963.g61996d73.el8.x86_64 12/25 2023-01-25T05:39:26.194 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libradosstriper1-2:18.0.0-1963.g61996d73.el8.x86_6 13/25 2023-01-25T05:39:26.279 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libradosstriper1-2:18.0.0-1963.g61996d73.el8.x86_6 13/25 2023-01-25T05:39:26.479 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 14/25 2023-01-25T05:39:27.281 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 15/25 2023-01-25T05:39:27.719 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librgw2-2:18.0.0-1963.g61996d73.el8.x86_64 16/25 2023-01-25T05:39:27.786 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librgw2-2:18.0.0-1963.g61996d73.el8.x86_64 16/25 2023-01-25T05:39:28.027 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rgw-2:18.0.0-1963.g61996d73.el8.x86_64 17/25 2023-01-25T05:39:28.229 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-01-25T05:39:28.435 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-common-2:18.0.0-1963.g61996d73.el8.x8 19/25 2023-01-25T05:39:32.199 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:18.0.0-1963.g61996d73.el8.x86_64 20/25 2023-01-25T05:39:32.435 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-common-2:18.0.0-1963.g61996d73.el8.x86_64 20/25 2023-01-25T05:39:33.025 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:18.0.0-1963.g61996d73.el8.x86_64 20/25 2023-01-25T05:39:33.405 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-base-2:18.0.0-1963.g61996d73.el8.x86_64 21/25 2023-01-25T05:39:33.493 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-base-2:18.0.0-1963.g61996d73.el8.x86_64 21/25 2023-01-25T05:39:33.668 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-selinux-2:18.0.0-1963.g61996d73.el8.x86_64 22/25 2023-01-25T05:39:52.832 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-selinux-2:18.0.0-1963.g61996d73.el8.x86_64 22/25 2023-01-25T05:39:52.832 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /sys 2023-01-25T05:39:52.832 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /proc 2023-01-25T05:39:52.833 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /dev 2023-01-25T05:39:52.833 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /run 2023-01-25T05:39:52.833 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /mnt 2023-01-25T05:39:52.833 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /var/tmp 2023-01-25T05:39:52.834 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /home 2023-01-25T05:39:52.834 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /tmp 2023-01-25T05:39:52.834 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /dev 2023-01-25T05:39:52.834 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:39:54.276 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-radosgw-2:18.0.0-1963.g61996d73.el8.x86_64 23/25 2023-01-25T05:39:54.296 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-1963.g61996d73.el8.x86_64 23/25 2023-01-25T05:39:54.297 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-01-25T05:39:54.525 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-01-25T05:39:54.525 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-01-25T05:39:55.236 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-01-25T05:39:55.236 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-base-2:18.0.0-1963.g61996d73.el8.x86_64 1/25 2023-01-25T05:39:55.236 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-common-2:18.0.0-1963.g61996d73.el8.x86_64 2/25 2023-01-25T05:39:55.237 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-radosgw-2:18.0.0-1963.g61996d73.el8.x86_64 3/25 2023-01-25T05:39:55.237 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-selinux-2:18.0.0-1963.g61996d73.el8.x86_64 4/25 2023-01-25T05:39:55.237 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs2-2:18.0.0-1963.g61996d73.el8.x86_64 5/25 2023-01-25T05:39:55.237 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libradosstriper1-2:18.0.0-1963.g61996d73.el8.x86_6 6/25 2023-01-25T05:39:55.237 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librgw2-2:18.0.0-1963.g61996d73.el8.x86_64 7/25 2023-01-25T05:39:55.238 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-argparse-2:18.0.0-1963.g61996d73.el8. 8/25 2023-01-25T05:39:55.238 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-common-2:18.0.0-1963.g61996d73.el8.x8 9/25 2023-01-25T05:39:55.238 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cephfs-2:18.0.0-1963.g61996d73.el8.x86_64 10/25 2023-01-25T05:39:55.238 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rados-2:18.0.0-1963.g61996d73.el8.x86_64 11/25 2023-01-25T05:39:55.239 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rbd-2:18.0.0-1963.g61996d73.el8.x86_64 12/25 2023-01-25T05:39:55.239 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rgw-2:18.0.0-1963.g61996d73.el8.x86_64 13/25 2023-01-25T05:39:55.239 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-01-25T05:39:55.239 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-01-25T05:39:55.239 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-01-25T05:39:55.240 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2023-01-25T05:39:55.240 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-01-25T05:39:55.240 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-01-25T05:39:55.240 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-01-25T05:39:55.240 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-01-25T05:39:55.241 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:18.0.0-1963.g61996d73.el8.x86_64 22/25 2023-01-25T05:39:55.241 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-01-25T05:39:55.241 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:18.0.0-1963.g61996d73.el8.x86_64 24/25 2023-01-25T05:39:56.099 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-01-25T05:39:56.100 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:39:56.100 INFO:teuthology.orchestra.run.smithi203.stdout:Upgraded: 2023-01-25T05:39:56.100 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:39:56.100 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:39:56.101 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2023-01-25T05:39:56.101 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:39:56.101 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:39:56.101 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:39:56.102 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:39:56.102 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-01-25T05:39:56.102 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:39:56.102 INFO:teuthology.orchestra.run.smithi203.stdout: liboath-2.6.2-3.el8.x86_64 2023-01-25T05:39:56.102 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-01-25T05:39:56.103 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:39:56.103 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-01-25T05:39:56.103 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:39:56.103 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind-1.3.1-3.el8.x86_64 2023-01-25T05:39:56.103 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-01-25T05:39:56.104 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:39:56.104 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:39:56.104 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:39:56.105 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-01-25T05:39:56.105 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:39:56.105 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:39:56.105 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:39:56.105 INFO:teuthology.orchestra.run.smithi203.stdout: thrift-0.13.0-2.el8.x86_64 2023-01-25T05:39:56.105 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:39:56.106 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:39:56.443 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-test 2023-01-25T05:39:57.180 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:43 ago on Wed 25 Jan 2023 05:39:14 AM UTC. 2023-01-25T05:39:58.331 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:39:58.332 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:39:58.332 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2023-01-25T05:39:58.333 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:39:58.333 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2023-01-25T05:39:58.333 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test x86_64 2:18.0.0-1963.g61996d73.el8 ceph 47 M 2023-01-25T05:39:58.333 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2023-01-25T05:39:58.334 INFO:teuthology.orchestra.run.smithi203.stdout: jq x86_64 1.6-6.el8 CentOS-AppStream 202 k 2023-01-25T05:39:58.334 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite x86_64 2:18.0.0-1963.g61996d73.el8 ceph 137 k 2023-01-25T05:39:58.334 INFO:teuthology.orchestra.run.smithi203.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2023-01-25T05:39:58.334 INFO:teuthology.orchestra.run.smithi203.stdout: socat x86_64 1.7.4.1-1.el8 CentOS-AppStream 323 k 2023-01-25T05:39:58.334 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet x86_64 1.6.1-20.el8 CentOS-AppStream 71 k 2023-01-25T05:39:58.335 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:39:58.335 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2023-01-25T05:39:58.335 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:39:58.335 INFO:teuthology.orchestra.run.smithi203.stdout:Install 6 Packages 2023-01-25T05:39:58.335 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:39:58.336 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 48 M 2023-01-25T05:39:58.336 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 218 M 2023-01-25T05:39:58.336 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2023-01-25T05:39:58.401 INFO:teuthology.orchestra.run.smithi203.stdout:(1/6): jq-1.6-6.el8.x86_64.rpm 3.0 MB/s | 202 kB 00:00 2023-01-25T05:39:58.426 INFO:teuthology.orchestra.run.smithi203.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 7.7 MB/s | 187 kB 00:00 2023-01-25T05:39:58.452 INFO:teuthology.orchestra.run.smithi203.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 13 MB/s | 323 kB 00:00 2023-01-25T05:39:58.477 INFO:teuthology.orchestra.run.smithi203.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 71 kB 00:00 2023-01-25T05:39:58.544 INFO:teuthology.orchestra.run.smithi203.stdout:(5/6): libcephsqlite-18.0.0-1963.g61996d73.el8. 657 kB/s | 137 kB 00:00 2023-01-25T05:40:00.644 INFO:teuthology.orchestra.run.smithi203.stdout:(6/6): ceph-test-18.0.0-1963.g61996d73.el8.x86_ 21 MB/s | 47 MB 00:02 2023-01-25T05:40:00.647 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2023-01-25T05:40:00.647 INFO:teuthology.orchestra.run.smithi203.stdout:Total 21 MB/s | 48 MB 00:02 2023-01-25T05:40:00.691 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2023-01-25T05:40:00.712 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2023-01-25T05:40:00.712 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2023-01-25T05:40:01.004 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2023-01-25T05:40:01.007 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2023-01-25T05:40:01.466 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2023-01-25T05:40:01.702 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-01-25T05:40:01.882 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-01-25T05:40:02.247 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-01-25T05:40:02.337 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-01-25T05:40:02.537 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : jq-1.6-6.el8.x86_64 4/6 2023-01-25T05:40:02.705 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephsqlite-2:18.0.0-1963.g61996d73.el8.x86_64 5/6 2023-01-25T05:40:07.317 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephsqlite-2:18.0.0-1963.g61996d73.el8.x86_64 5/6 2023-01-25T05:40:07.592 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-test-2:18.0.0-1963.g61996d73.el8.x86_64 6/6 2023-01-25T05:40:07.905 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-test-2:18.0.0-1963.g61996d73.el8.x86_64 6/6 2023-01-25T05:40:07.905 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-test-2:18.0.0-1963.g61996d73.el8.x86_64 1/6 2023-01-25T05:40:07.906 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephsqlite-2:18.0.0-1963.g61996d73.el8.x86_64 2/6 2023-01-25T05:40:07.906 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : jq-1.6-6.el8.x86_64 3/6 2023-01-25T05:40:07.906 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-01-25T05:40:07.906 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-01-25T05:40:08.448 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-01-25T05:40:08.448 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:40:08.448 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2023-01-25T05:40:08.449 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:40:08.449 INFO:teuthology.orchestra.run.smithi203.stdout: jq-1.6-6.el8.x86_64 2023-01-25T05:40:08.449 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:40:08.449 INFO:teuthology.orchestra.run.smithi203.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-01-25T05:40:08.450 INFO:teuthology.orchestra.run.smithi203.stdout: socat-1.7.4.1-1.el8.x86_64 2023-01-25T05:40:08.450 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-01-25T05:40:08.450 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:40:08.450 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:40:08.654 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph 2023-01-25T05:40:09.387 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:55 ago on Wed 25 Jan 2023 05:39:14 AM UTC. 2023-01-25T05:40:10.540 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:40:10.543 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2023-01-25T05:40:10.543 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2023-01-25T05:40:10.544 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2023-01-25T05:40:10.544 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2023-01-25T05:40:10.544 INFO:teuthology.orchestra.run.smithi203.stdout: ceph x86_64 2:18.0.0-1963.g61996d73.el8 ceph 6.4 k 2023-01-25T05:40:10.544 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2023-01-25T05:40:10.545 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds x86_64 2:18.0.0-1963.g61996d73.el8 ceph 2.3 M 2023-01-25T05:40:10.545 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr x86_64 2:18.0.0-1963.g61996d73.el8 ceph 1.5 M 2023-01-25T05:40:10.545 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core noarch 2:18.0.0-1963.g61996d73.el8 ceph-noarch 236 k 2023-01-25T05:40:10.546 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon x86_64 2:18.0.0-1963.g61996d73.el8 ceph 3.9 M 2023-01-25T05:40:10.546 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd x86_64 2:18.0.0-1963.g61996d73.el8 ceph 17 M 2023-01-25T05:40:10.546 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-01-25T05:40:10.546 INFO:teuthology.orchestra.run.smithi203.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-01-25T05:40:10.546 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-BaseOS 237 k 2023-01-25T05:40:10.547 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-01-25T05:40:10.547 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-01-25T05:40:10.547 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography x86_64 3.2.1-5.el8 CentOS-BaseOS 559 k 2023-01-25T05:40:10.547 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-01-25T05:40:10.547 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-01-25T05:40:10.548 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-01-25T05:40:10.548 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2023-01-25T05:40:10.548 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2023-01-25T05:40:10.548 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-01-25T05:40:10.549 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-01-25T05:40:10.549 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-01-25T05:40:10.549 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2023-01-25T05:40:10.549 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-BaseOS 109 k 2023-01-25T05:40:10.550 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2023-01-25T05:40:10.550 INFO:teuthology.orchestra.run.smithi203.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-01-25T05:40:10.550 INFO:teuthology.orchestra.run.smithi203.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-01-25T05:40:10.550 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-01-25T05:40:10.550 INFO:teuthology.orchestra.run.smithi203.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-01-25T05:40:10.551 INFO:teuthology.orchestra.run.smithi203.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-01-25T05:40:10.551 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-01-25T05:40:10.551 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-01-25T05:40:10.551 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2023-01-25T05:40:10.551 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-01-25T05:40:10.552 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:40:10.552 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2023-01-25T05:40:10.552 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2023-01-25T05:40:10.552 INFO:teuthology.orchestra.run.smithi203.stdout:Install 32 Packages 2023-01-25T05:40:10.552 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:40:10.553 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 28 M 2023-01-25T05:40:10.553 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 110 M 2023-01-25T05:40:10.554 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2023-01-25T05:40:10.687 INFO:teuthology.orchestra.run.smithi203.stdout:(1/32): ceph-18.0.0-1963.g61996d73.el8.x86_64.r 46 kB/s | 6.4 kB 00:00 2023-01-25T05:40:10.854 INFO:teuthology.orchestra.run.smithi203.stdout:(2/32): ceph-mgr-18.0.0-1963.g61996d73.el8.x86_ 5.0 MB/s | 1.5 MB 00:00 2023-01-25T05:40:10.979 INFO:teuthology.orchestra.run.smithi203.stdout:(3/32): ceph-mds-18.0.0-1963.g61996d73.el8.x86_ 5.3 MB/s | 2.3 MB 00:00 2023-01-25T05:40:11.071 INFO:teuthology.orchestra.run.smithi203.stdout:(4/32): ceph-mgr-modules-core-18.0.0-1963.g6199 2.5 MB/s | 236 kB 00:00 2023-01-25T05:40:11.121 INFO:teuthology.orchestra.run.smithi203.stdout:(5/32): python3-mako-1.0.6-13.el8.noarch.rpm 3.1 MB/s | 157 kB 00:00 2023-01-25T05:40:11.147 INFO:teuthology.orchestra.run.smithi203.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 1.5 MB/s | 39 kB 00:00 2023-01-25T05:40:11.172 INFO:teuthology.orchestra.run.smithi203.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 4.0 MB/s | 103 kB 00:00 2023-01-25T05:40:11.197 INFO:teuthology.orchestra.run.smithi203.stdout:(8/32): python3-pytz-2017.2-9.el8.noarch.rpm 2.1 MB/s | 54 kB 00:00 2023-01-25T05:40:11.289 INFO:teuthology.orchestra.run.smithi203.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 4.9 MB/s | 457 kB 00:00 2023-01-25T05:40:11.348 INFO:teuthology.orchestra.run.smithi203.stdout:(10/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 4.0 MB/s | 237 kB 00:00 2023-01-25T05:40:11.382 INFO:teuthology.orchestra.run.smithi203.stdout:(11/32): python3-cryptography-3.2.1-5.el8.x86_6 16 MB/s | 559 kB 00:00 2023-01-25T05:40:11.407 INFO:teuthology.orchestra.run.smithi203.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 4.3 MB/s | 109 kB 00:00 2023-01-25T05:40:11.757 INFO:teuthology.orchestra.run.smithi203.stdout:(13/32): ceph-osd-18.0.0-1963.g61996d73.el8.x86 19 MB/s | 17 MB 00:00 2023-01-25T05:40:11.774 INFO:teuthology.orchestra.run.smithi203.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 119 kB/s | 44 kB 00:00 2023-01-25T05:40:11.924 INFO:teuthology.orchestra.run.smithi203.stdout:(15/32): ceph-mon-18.0.0-1963.g61996d73.el8.x86 3.1 MB/s | 3.9 MB 00:01 2023-01-25T05:40:11.950 INFO:teuthology.orchestra.run.smithi203.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 984 kB/s | 173 kB 00:00 2023-01-25T05:40:11.975 INFO:teuthology.orchestra.run.smithi203.stdout:(17/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 853 kB/s | 185 kB 00:00 2023-01-25T05:40:11.992 INFO:teuthology.orchestra.run.smithi203.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 264 kB/s | 11 kB 00:00 2023-01-25T05:40:12.026 INFO:teuthology.orchestra.run.smithi203.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 348 kB/s | 18 kB 00:00 2023-01-25T05:40:12.051 INFO:teuthology.orchestra.run.smithi203.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 831 kB/s | 49 kB 00:00 2023-01-25T05:40:12.077 INFO:teuthology.orchestra.run.smithi203.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2023-01-25T05:40:12.129 INFO:teuthology.orchestra.run.smithi203.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.9 MB/s | 384 kB 00:00 2023-01-25T05:40:12.152 INFO:teuthology.orchestra.run.smithi203.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 206 kB/s | 16 kB 00:00 2023-01-25T05:40:12.177 INFO:teuthology.orchestra.run.smithi203.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 372 kB/s | 19 kB 00:00 2023-01-25T05:40:12.202 INFO:teuthology.orchestra.run.smithi203.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 476 kB/s | 24 kB 00:00 2023-01-25T05:40:12.228 INFO:teuthology.orchestra.run.smithi203.stdout:(26/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.6 MB/s | 283 kB 00:00 2023-01-25T05:40:12.253 INFO:teuthology.orchestra.run.smithi203.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 386 kB/s | 29 kB 00:00 2023-01-25T05:40:12.279 INFO:teuthology.orchestra.run.smithi203.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 364 kB/s | 27 kB 00:00 2023-01-25T05:40:12.304 INFO:teuthology.orchestra.run.smithi203.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 4.9 MB/s | 251 kB 00:00 2023-01-25T05:40:12.354 INFO:teuthology.orchestra.run.smithi203.stdout:(30/32): python3-zc-lockfile-2.0-2.el8.noarch.r 459 kB/s | 23 kB 00:00 2023-01-25T05:40:12.379 INFO:teuthology.orchestra.run.smithi203.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 850 kB/s | 85 kB 00:00 2023-01-25T05:40:12.404 INFO:teuthology.orchestra.run.smithi203.stdout:(32/32): python3-waitress-1.2.1-2.el8.1.noarch. 1.3 MB/s | 241 kB 00:00 2023-01-25T05:40:12.407 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2023-01-25T05:40:12.407 INFO:teuthology.orchestra.run.smithi203.stdout:Total 15 MB/s | 28 MB 00:01 2023-01-25T05:40:12.485 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2023-01-25T05:40:12.522 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2023-01-25T05:40:12.522 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2023-01-25T05:40:12.909 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2023-01-25T05:40:12.912 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2023-01-25T05:40:14.129 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2023-01-25T05:40:14.376 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-01-25T05:40:14.636 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-01-25T05:40:14.918 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-01-25T05:40:15.193 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/32 2023-01-25T05:40:15.382 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-01-25T05:40:15.554 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-01-25T05:40:15.738 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-01-25T05:40:15.908 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-01-25T05:40:16.140 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-01-25T05:40:16.419 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-01-25T05:40:16.673 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-01-25T05:40:16.913 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-01-25T05:40:17.358 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 13/32 2023-01-25T05:40:17.623 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 14/32 2023-01-25T05:40:17.843 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-01-25T05:40:18.102 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-01-25T05:40:18.345 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-01-25T05:40:18.671 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-01-25T05:40:18.916 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-01-25T05:40:19.129 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 20/32 2023-01-25T05:40:19.340 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-01-25T05:40:19.609 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-01-25T05:40:19.864 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-01-25T05:40:20.096 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-01-25T05:40:20.467 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 25/32 2023-01-25T05:40:20.840 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-01-25T05:40:21.199 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-1963.g61996d73.el8. 27/32 2023-01-25T05:40:21.413 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-2:18.0.0-1963.g61996d73.el8.x86_64 28/32 2023-01-25T05:40:22.919 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-2:18.0.0-1963.g61996d73.el8.x86_64 28/32 2023-01-25T05:40:23.121 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-osd-2:18.0.0-1963.g61996d73.el8.x86_64 29/32 2023-01-25T05:40:23.519 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-osd-2:18.0.0-1963.g61996d73.el8.x86_64 29/32 2023-01-25T05:40:23.718 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mon-2:18.0.0-1963.g61996d73.el8.x86_64 30/32 2023-01-25T05:40:23.933 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mon-2:18.0.0-1963.g61996d73.el8.x86_64 30/32 2023-01-25T05:40:24.126 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mds-2:18.0.0-1963.g61996d73.el8.x86_64 31/32 2023-01-25T05:40:24.144 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mds-2:18.0.0-1963.g61996d73.el8.x86_64 31/32 2023-01-25T05:40:24.250 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-2:18.0.0-1963.g61996d73.el8.x86_64 32/32 2023-01-25T05:40:24.698 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-2:18.0.0-1963.g61996d73.el8.x86_64 32/32 2023-01-25T05:40:24.698 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-2:18.0.0-1963.g61996d73.el8.x86_64 1/32 2023-01-25T05:40:24.698 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mds-2:18.0.0-1963.g61996d73.el8.x86_64 2/32 2023-01-25T05:40:24.698 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-2:18.0.0-1963.g61996d73.el8.x86_64 3/32 2023-01-25T05:40:24.699 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mon-2:18.0.0-1963.g61996d73.el8.x86_64 4/32 2023-01-25T05:40:24.699 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-osd-2:18.0.0-1963.g61996d73.el8.x86_64 5/32 2023-01-25T05:40:24.699 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-1963.g61996d73.el8. 6/32 2023-01-25T05:40:24.699 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 7/32 2023-01-25T05:40:24.699 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-01-25T05:40:24.700 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-01-25T05:40:24.700 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 10/32 2023-01-25T05:40:24.700 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-01-25T05:40:24.700 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 12/32 2023-01-25T05:40:24.700 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 13/32 2023-01-25T05:40:24.701 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-01-25T05:40:24.701 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-01-25T05:40:24.701 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-01-25T05:40:24.701 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-01-25T05:40:24.701 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-01-25T05:40:24.702 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-01-25T05:40:24.702 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-01-25T05:40:24.702 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-01-25T05:40:24.702 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-01-25T05:40:24.702 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-01-25T05:40:24.703 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-01-25T05:40:24.703 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-01-25T05:40:24.703 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-01-25T05:40:24.703 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-01-25T05:40:24.703 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-01-25T05:40:24.704 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 29/32 2023-01-25T05:40:24.704 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-01-25T05:40:24.704 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-01-25T05:40:25.633 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-01-25T05:40:25.634 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:40:25.634 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2023-01-25T05:40:25.634 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:40:25.634 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:40:25.634 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:40:25.635 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core-2:18.0.0-1963.g61996d73.el8.noarch 2023-01-25T05:40:25.635 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:40:25.635 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:40:25.635 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-01-25T05:40:25.635 INFO:teuthology.orchestra.run.smithi203.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-01-25T05:40:25.636 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-01-25T05:40:25.636 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-01-25T05:40:25.636 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-01-25T05:40:25.636 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-01-25T05:40:25.636 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-6.2-6.el8.noarch 2023-01-25T05:40:25.637 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-01-25T05:40:25.637 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-01-25T05:40:25.637 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako-1.0.6-13.el8.noarch 2023-01-25T05:40:25.637 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-01-25T05:40:25.637 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-01-25T05:40:25.638 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-01-25T05:40:25.638 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend-2.6-1.el8.noarch 2023-01-25T05:40:25.638 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-01-25T05:40:25.638 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser-2.14-14.el8.noarch 2023-01-25T05:40:25.639 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz-2017.2-9.el8.noarch 2023-01-25T05:40:25.639 INFO:teuthology.orchestra.run.smithi203.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-01-25T05:40:25.639 INFO:teuthology.orchestra.run.smithi203.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-01-25T05:40:25.639 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-01-25T05:40:25.640 INFO:teuthology.orchestra.run.smithi203.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-01-25T05:40:25.640 INFO:teuthology.orchestra.run.smithi203.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-01-25T05:40:25.640 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-01-25T05:40:25.640 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-01-25T05:40:25.640 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-01-25T05:40:25.641 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-01-25T05:40:25.641 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:40:25.641 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:40:25.795 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-base 2023-01-25T05:40:26.519 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:12 ago on Wed 25 Jan 2023 05:39:14 AM UTC. 2023-01-25T05:40:27.559 INFO:teuthology.orchestra.run.smithi203.stdout:Package ceph-base-2:18.0.0-1963.g61996d73.el8.x86_64 is already installed. 2023-01-25T05:40:27.660 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:40:27.662 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2023-01-25T05:40:27.662 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:40:28.085 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install cephadm 2023-01-25T05:40:28.805 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:14 ago on Wed 25 Jan 2023 05:39:14 AM UTC. 2023-01-25T05:40:29.946 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:40:29.947 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:40:29.947 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2023-01-25T05:40:29.948 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:40:29.948 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2023-01-25T05:40:29.948 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm noarch 2:18.0.0-1963.g61996d73.el8 ceph-noarch 82 k 2023-01-25T05:40:29.948 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:40:29.949 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2023-01-25T05:40:29.949 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:40:29.949 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2023-01-25T05:40:29.949 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:40:29.949 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 82 k 2023-01-25T05:40:29.950 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 374 k 2023-01-25T05:40:29.950 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2023-01-25T05:40:30.118 INFO:teuthology.orchestra.run.smithi203.stdout:cephadm-18.0.0-1963.g61996d73.el8.noarch.rpm 484 kB/s | 82 kB 00:00 2023-01-25T05:40:30.119 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2023-01-25T05:40:30.119 INFO:teuthology.orchestra.run.smithi203.stdout:Total 480 kB/s | 82 kB 00:00 2023-01-25T05:40:30.120 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2023-01-25T05:40:30.127 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2023-01-25T05:40:30.127 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2023-01-25T05:40:30.129 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2023-01-25T05:40:30.133 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2023-01-25T05:40:30.180 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2023-01-25T05:40:31.749 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:18.0.0-1963.g61996d73.el8.noarch 1/1 2023-01-25T05:40:31.903 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephadm-2:18.0.0-1963.g61996d73.el8.noarch 1/1 2023-01-25T05:40:32.142 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:18.0.0-1963.g61996d73.el8.noarch 1/1 2023-01-25T05:40:32.601 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephadm-2:18.0.0-1963.g61996d73.el8.noarch 1/1 2023-01-25T05:40:32.601 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:40:32.602 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2023-01-25T05:40:32.602 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm-2:18.0.0-1963.g61996d73.el8.noarch 2023-01-25T05:40:32.602 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:40:32.602 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:40:32.778 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-immutable-object-cache 2023-01-25T05:40:33.519 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:19 ago on Wed 25 Jan 2023 05:39:14 AM UTC. 2023-01-25T05:40:34.674 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:40:34.675 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:40:34.675 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2023-01-25T05:40:34.675 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:40:34.675 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2023-01-25T05:40:34.676 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-1963.g61996d73.el8 ceph 165 k 2023-01-25T05:40:34.676 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:40:34.676 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2023-01-25T05:40:34.676 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:40:34.676 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2023-01-25T05:40:34.677 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:40:34.677 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 165 k 2023-01-25T05:40:34.677 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 503 k 2023-01-25T05:40:34.677 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2023-01-25T05:40:35.074 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-immutable-object-cache-18.0.0-1963.g61996d 416 kB/s | 165 kB 00:00 2023-01-25T05:40:35.075 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2023-01-25T05:40:35.075 INFO:teuthology.orchestra.run.smithi203.stdout:Total 414 kB/s | 165 kB 00:00 2023-01-25T05:40:35.075 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2023-01-25T05:40:35.086 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2023-01-25T05:40:35.086 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2023-01-25T05:40:35.135 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2023-01-25T05:40:35.138 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2023-01-25T05:40:35.254 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2023-01-25T05:40:35.433 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-1963.g61996d73. 1/1 2023-01-25T05:40:35.904 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-1963.g61996d73. 1/1 2023-01-25T05:40:36.374 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-1963.g61996d73. 1/1 2023-01-25T05:40:36.375 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:40:36.375 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2023-01-25T05:40:36.375 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:40:36.375 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:40:36.375 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:40:36.566 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr 2023-01-25T05:40:37.299 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:23 ago on Wed 25 Jan 2023 05:39:14 AM UTC. 2023-01-25T05:40:38.353 INFO:teuthology.orchestra.run.smithi203.stdout:Package ceph-mgr-2:18.0.0-1963.g61996d73.el8.x86_64 is already installed. 2023-01-25T05:40:38.453 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:40:38.454 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2023-01-25T05:40:38.455 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:40:38.533 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-dashboard 2023-01-25T05:40:39.258 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:25 ago on Wed 25 Jan 2023 05:39:14 AM UTC. 2023-01-25T05:40:40.394 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:40:40.395 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:40:40.395 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2023-01-25T05:40:40.395 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:40:40.396 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2023-01-25T05:40:40.396 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard noarch 2:18.0.0-1963.g61996d73.el8 ceph-noarch 1.7 M 2023-01-25T05:40:40.396 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2023-01-25T05:40:40.396 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards noarch 2:18.0.0-1963.g61996d73.el8 ceph-noarch 23 k 2023-01-25T05:40:40.397 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts noarch 2:18.0.0-1963.g61996d73.el8 ceph-noarch 15 k 2023-01-25T05:40:40.397 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-BaseOS 43 k 2023-01-25T05:40:40.397 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-01-25T05:40:40.397 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-01-25T05:40:40.397 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:40:40.398 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2023-01-25T05:40:40.398 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:40:40.398 INFO:teuthology.orchestra.run.smithi203.stdout:Install 6 Packages 2023-01-25T05:40:40.398 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:40:40.399 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 2.0 M 2023-01-25T05:40:40.399 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 8.2 M 2023-01-25T05:40:40.399 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2023-01-25T05:40:40.553 INFO:teuthology.orchestra.run.smithi203.stdout:(1/6): ceph-prometheus-alerts-18.0.0-1963.g6199 94 kB/s | 15 kB 00:00 2023-01-25T05:40:40.570 INFO:teuthology.orchestra.run.smithi203.stdout:(2/6): ceph-grafana-dashboards-18.0.0-1963.g619 134 kB/s | 23 kB 00:00 2023-01-25T05:40:40.620 INFO:teuthology.orchestra.run.smithi203.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 637 kB/s | 43 kB 00:00 2023-01-25T05:40:40.670 INFO:teuthology.orchestra.run.smithi203.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 335 kB/s | 34 kB 00:00 2023-01-25T05:40:40.706 INFO:teuthology.orchestra.run.smithi203.stdout:(5/6): ceph-mgr-dashboard-18.0.0-1963.g61996d73 5.6 MB/s | 1.7 MB 00:00 2023-01-25T05:40:40.782 INFO:teuthology.orchestra.run.smithi203.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.2 MB/s | 196 kB 00:00 2023-01-25T05:40:40.784 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2023-01-25T05:40:40.784 INFO:teuthology.orchestra.run.smithi203.stdout:Total 5.2 MB/s | 2.0 MB 00:00 2023-01-25T05:40:40.794 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2023-01-25T05:40:40.802 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2023-01-25T05:40:40.802 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2023-01-25T05:40:40.883 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2023-01-25T05:40:40.887 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2023-01-25T05:40:41.166 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2023-01-25T05:40:41.429 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-01-25T05:40:41.649 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-01-25T05:40:41.827 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-01-25T05:40:41.946 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-1963.g61996d73.el8.n 4/6 2023-01-25T05:40:42.303 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-1963.g61996d73.el8. 5/6 2023-01-25T05:40:42.551 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-1963.g61996d73.el8.noarc 6/6 2023-01-25T05:40:42.858 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-1963.g61996d73.el8.noarc 6/6 2023-01-25T05:40:42.859 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-1963.g61996d73.el8. 1/6 2023-01-25T05:40:42.859 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-1963.g61996d73.el8.noarc 2/6 2023-01-25T05:40:42.859 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-1963.g61996d73.el8.n 3/6 2023-01-25T05:40:42.859 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-01-25T05:40:42.860 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-01-25T05:40:43.349 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-01-25T05:40:43.349 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:40:43.349 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2023-01-25T05:40:43.349 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards-2:18.0.0-1963.g61996d73.el8.noarch 2023-01-25T05:40:43.350 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard-2:18.0.0-1963.g61996d73.el8.noarch 2023-01-25T05:40:43.350 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts-2:18.0.0-1963.g61996d73.el8.noarch 2023-01-25T05:40:43.350 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-01-25T05:40:43.350 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-01-25T05:40:43.350 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes-2.4.1-12.el8.noarch 2023-01-25T05:40:43.351 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:40:43.351 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:40:43.461 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-diskprediction-local 2023-01-25T05:40:44.182 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:30 ago on Wed 25 Jan 2023 05:39:14 AM UTC. 2023-01-25T05:40:45.325 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:40:45.327 INFO:teuthology.orchestra.run.smithi203.stdout:===================================================================================================== 2023-01-25T05:40:45.327 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2023-01-25T05:40:45.327 INFO:teuthology.orchestra.run.smithi203.stdout:===================================================================================================== 2023-01-25T05:40:45.327 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2023-01-25T05:40:45.328 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-1963.g61996d73.el8 ceph-noarch 7.4 M 2023-01-25T05:40:45.328 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2023-01-25T05:40:45.328 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran x86_64 8.5.0-18.el8 CentOS-BaseOS 645 k 2023-01-25T05:40:45.328 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath x86_64 8.5.0-18.el8 CentOS-BaseOS 172 k 2023-01-25T05:40:45.328 INFO:teuthology.orchestra.run.smithi203.stdout: openblas x86_64 0.3.15-4.el8 CentOS-AppStream 4.7 M 2023-01-25T05:40:45.329 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-threads x86_64 0.3.15-4.el8 CentOS-AppStream 4.9 M 2023-01-25T05:40:45.329 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy x86_64 1:1.14.3-10.el8 CentOS-AppStream 3.7 M 2023-01-25T05:40:45.329 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 CentOS-AppStream 225 k 2023-01-25T05:40:45.329 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 CentOS-AppStream 14 M 2023-01-25T05:40:45.329 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:40:45.330 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2023-01-25T05:40:45.330 INFO:teuthology.orchestra.run.smithi203.stdout:===================================================================================================== 2023-01-25T05:40:45.330 INFO:teuthology.orchestra.run.smithi203.stdout:Install 8 Packages 2023-01-25T05:40:45.330 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:40:45.331 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 35 M 2023-01-25T05:40:45.331 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 210 M 2023-01-25T05:40:45.331 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2023-01-25T05:40:45.529 INFO:teuthology.orchestra.run.smithi203.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 24 MB/s | 4.7 MB 00:00 2023-01-25T05:40:45.596 INFO:teuthology.orchestra.run.smithi203.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 18 MB/s | 4.9 MB 00:00 2023-01-25T05:40:45.630 INFO:teuthology.orchestra.run.smithi203.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 6.6 MB/s | 225 kB 00:00 2023-01-25T05:40:45.722 INFO:teuthology.orchestra.run.smithi203.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 19 MB/s | 3.7 MB 00:00 2023-01-25T05:40:45.764 INFO:teuthology.orchestra.run.smithi203.stdout:(5/8): libgfortran-8.5.0-18.el8.x86_64.rpm 15 MB/s | 645 kB 00:00 2023-01-25T05:40:45.797 INFO:teuthology.orchestra.run.smithi203.stdout:(6/8): libquadmath-8.5.0-18.el8.x86_64.rpm 5.1 MB/s | 172 kB 00:00 2023-01-25T05:40:46.031 INFO:teuthology.orchestra.run.smithi203.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 34 MB/s | 14 MB 00:00 2023-01-25T05:40:46.131 INFO:teuthology.orchestra.run.smithi203.stdout:(8/8): ceph-mgr-diskprediction-local-18.0.0-196 9.2 MB/s | 7.4 MB 00:00 2023-01-25T05:40:46.133 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2023-01-25T05:40:46.134 INFO:teuthology.orchestra.run.smithi203.stdout:Total 44 MB/s | 35 MB 00:00 2023-01-25T05:40:46.364 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2023-01-25T05:40:46.383 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2023-01-25T05:40:46.383 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2023-01-25T05:40:46.781 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2023-01-25T05:40:46.784 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2023-01-25T05:40:47.353 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2023-01-25T05:40:47.505 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libquadmath-8.5.0-18.el8.x86_64 1/8 2023-01-25T05:40:47.639 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libquadmath-8.5.0-18.el8.x86_64 1/8 2023-01-25T05:40:47.780 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libgfortran-8.5.0-18.el8.x86_64 2/8 2023-01-25T05:40:48.393 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libgfortran-8.5.0-18.el8.x86_64 2/8 2023-01-25T05:40:48.589 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-01-25T05:40:49.222 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-01-25T05:40:49.422 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-01-25T05:40:50.124 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-01-25T05:40:50.694 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-01-25T05:40:50.922 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-01-25T05:40:52.712 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-01-25T05:40:53.987 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-01-25T05:40:54.180 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-1963.g61996d7 8/8 2023-01-25T05:40:54.513 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-1963.g61996d7 8/8 2023-01-25T05:40:54.514 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-1963.g61996d7 1/8 2023-01-25T05:40:54.514 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-01-25T05:40:54.515 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-01-25T05:40:54.515 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-01-25T05:40:54.516 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-01-25T05:40:54.516 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-01-25T05:40:54.516 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libgfortran-8.5.0-18.el8.x86_64 7/8 2023-01-25T05:40:55.025 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libquadmath-8.5.0-18.el8.x86_64 8/8 2023-01-25T05:40:55.026 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:40:55.026 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2023-01-25T05:40:55.026 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local-2:18.0.0-1963.g61996d73.el8.noarch 2023-01-25T05:40:55.027 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran-8.5.0-18.el8.x86_64 2023-01-25T05:40:55.027 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath-8.5.0-18.el8.x86_64 2023-01-25T05:40:55.027 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-0.3.15-4.el8.x86_64 2023-01-25T05:40:55.028 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-01-25T05:40:55.028 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-01-25T05:40:55.028 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-01-25T05:40:55.029 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-01-25T05:40:55.029 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:40:55.030 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:40:55.131 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-rook 2023-01-25T05:40:55.857 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:41 ago on Wed 25 Jan 2023 05:39:14 AM UTC. 2023-01-25T05:40:57.005 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:40:57.007 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2023-01-25T05:40:57.007 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2023-01-25T05:40:57.007 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2023-01-25T05:40:57.008 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2023-01-25T05:40:57.008 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook noarch 2:18.0.0-1963.g61996d73.el8 ceph-noarch 47 k 2023-01-25T05:40:57.008 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2023-01-25T05:40:57.008 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-01-25T05:40:57.009 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-01-25T05:40:57.009 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-01-25T05:40:57.009 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2023-01-25T05:40:57.009 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2023-01-25T05:40:57.009 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-01-25T05:40:57.010 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-BaseOS 155 k 2023-01-25T05:40:57.010 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2023-01-25T05:40:57.010 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2023-01-25T05:40:57.011 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-BaseOS 43 k 2023-01-25T05:40:57.011 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-01-25T05:40:57.011 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-01-25T05:40:57.012 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:40:57.012 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2023-01-25T05:40:57.012 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2023-01-25T05:40:57.013 INFO:teuthology.orchestra.run.smithi203.stdout:Install 13 Packages 2023-01-25T05:40:57.013 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:40:57.014 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 1.9 M 2023-01-25T05:40:57.015 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 25 M 2023-01-25T05:40:57.015 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2023-01-25T05:40:57.080 INFO:teuthology.orchestra.run.smithi203.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 382 kB/s | 27 kB 00:00 2023-01-25T05:40:57.105 INFO:teuthology.orchestra.run.smithi203.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 211 kB/s | 20 kB 00:00 2023-01-25T05:40:57.131 INFO:teuthology.orchestra.run.smithi203.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2023-01-25T05:40:57.156 INFO:teuthology.orchestra.run.smithi203.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.1 MB/s | 110 kB 00:00 2023-01-25T05:40:57.182 INFO:teuthology.orchestra.run.smithi203.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 3.0 MB/s | 155 kB 00:00 2023-01-25T05:40:57.207 INFO:teuthology.orchestra.run.smithi203.stdout:(6/13): ceph-mgr-rook-18.0.0-1963.g61996d73.el8 241 kB/s | 47 kB 00:00 2023-01-25T05:40:57.241 INFO:teuthology.orchestra.run.smithi203.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 505 kB/s | 43 kB 00:00 2023-01-25T05:40:57.316 INFO:teuthology.orchestra.run.smithi203.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 149 kB/s | 16 kB 00:00 2023-01-25T05:40:57.341 INFO:teuthology.orchestra.run.smithi203.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 207 kB/s | 33 kB 00:00 2023-01-25T05:40:57.375 INFO:teuthology.orchestra.run.smithi203.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 608 kB/s | 81 kB 00:00 2023-01-25T05:40:57.425 INFO:teuthology.orchestra.run.smithi203.stdout:(11/13): python3-rsa-4.8-1.el8.noarch.rpm 735 kB/s | 61 kB 00:00 2023-01-25T05:40:57.450 INFO:teuthology.orchestra.run.smithi203.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 813 kB/s | 61 kB 00:00 2023-01-25T05:40:57.542 INFO:teuthology.orchestra.run.smithi203.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.2 MB/s | 1.2 MB 00:00 2023-01-25T05:40:57.544 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2023-01-25T05:40:57.544 INFO:teuthology.orchestra.run.smithi203.stdout:Total 3.6 MB/s | 1.9 MB 00:00 2023-01-25T05:40:57.597 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2023-01-25T05:40:57.612 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2023-01-25T05:40:57.612 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2023-01-25T05:40:57.711 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2023-01-25T05:40:57.715 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2023-01-25T05:40:58.216 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2023-01-25T05:40:58.465 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-01-25T05:40:58.663 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-01-25T05:40:58.906 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rsa-4.8-1.el8.noarch 3/13 2023-01-25T05:40:59.127 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-01-25T05:40:59.334 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-01-25T05:40:59.573 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-01-25T05:40:59.907 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-01-25T05:41:00.133 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-01-25T05:41:01.183 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-01-25T05:41:01.460 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-01-25T05:41:01.677 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-01-25T05:41:01.862 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-01-25T05:41:02.445 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-rook-2:18.0.0-1963.g61996d73.el8.noarch 13/13 2023-01-25T05:41:02.765 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-1963.g61996d73.el8.noarch 13/13 2023-01-25T05:41:02.765 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-rook-2:18.0.0-1963.g61996d73.el8.noarch 1/13 2023-01-25T05:41:02.765 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-01-25T05:41:02.766 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-01-25T05:41:02.766 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-01-25T05:41:02.766 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-01-25T05:41:02.766 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-01-25T05:41:02.766 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-01-25T05:41:02.766 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-01-25T05:41:02.767 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-01-25T05:41:02.767 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-01-25T05:41:02.767 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-01-25T05:41:02.767 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 12/13 2023-01-25T05:41:03.323 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-01-25T05:41:03.323 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:03.323 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2023-01-25T05:41:03.324 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook-2:18.0.0-1963.g61996d73.el8.noarch 2023-01-25T05:41:03.324 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-01-25T05:41:03.324 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-01-25T05:41:03.324 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-01-25T05:41:03.325 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-01-25T05:41:03.325 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-01-25T05:41:03.325 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-01-25T05:41:03.325 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-01-25T05:41:03.325 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-01-25T05:41:03.326 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-01-25T05:41:03.326 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-01-25T05:41:03.326 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa-4.8-1.el8.noarch 2023-01-25T05:41:03.326 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-01-25T05:41:03.326 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:03.327 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:41:03.457 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-cephadm 2023-01-25T05:41:04.186 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:50 ago on Wed 25 Jan 2023 05:39:14 AM UTC. 2023-01-25T05:41:05.325 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:41:05.325 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:41:05.325 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2023-01-25T05:41:05.326 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:41:05.326 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2023-01-25T05:41:05.326 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm noarch 2:18.0.0-1963.g61996d73.el8 ceph-noarch 120 k 2023-01-25T05:41:05.326 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2023-01-25T05:41:05.326 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-01-25T05:41:05.327 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel noarch 2.5.1-7.el8 CentOS-AppStream 4.8 M 2023-01-25T05:41:05.327 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2 noarch 2.10.1-3.el8 CentOS-AppStream 538 k 2023-01-25T05:41:05.327 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-01-25T05:41:05.327 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:05.327 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2023-01-25T05:41:05.328 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:41:05.328 INFO:teuthology.orchestra.run.smithi203.stdout:Install 5 Packages 2023-01-25T05:41:05.328 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:05.329 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 6.0 M 2023-01-25T05:41:05.329 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 26 M 2023-01-25T05:41:05.329 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2023-01-25T05:41:05.433 INFO:teuthology.orchestra.run.smithi203.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 5.0 MB/s | 538 kB 00:00 2023-01-25T05:41:05.509 INFO:teuthology.orchestra.run.smithi203.stdout:(2/5): python3-babel-2.5.1-7.el8.noarch.rpm 27 MB/s | 4.8 MB 00:00 2023-01-25T05:41:05.534 INFO:teuthology.orchestra.run.smithi203.stdout:(3/5): ceph-mgr-cephadm-18.0.0-1963.g61996d73.e 583 kB/s | 120 kB 00:00 2023-01-25T05:41:05.642 INFO:teuthology.orchestra.run.smithi203.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 451 kB/s | 60 kB 00:00 2023-01-25T05:41:05.726 INFO:teuthology.orchestra.run.smithi203.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.6 MB/s | 476 kB 00:00 2023-01-25T05:41:05.728 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2023-01-25T05:41:05.728 INFO:teuthology.orchestra.run.smithi203.stdout:Total 15 MB/s | 6.0 MB 00:00 2023-01-25T05:41:05.779 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2023-01-25T05:41:05.790 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2023-01-25T05:41:05.790 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2023-01-25T05:41:05.934 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2023-01-25T05:41:05.937 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2023-01-25T05:41:06.218 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2023-01-25T05:41:06.517 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-01-25T05:41:07.524 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-01-25T05:41:08.230 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-01-25T05:41:08.480 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-01-25T05:41:08.677 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-1963.g61996d73.el8.noarch 5/5 2023-01-25T05:41:08.970 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-1963.g61996d73.el8.noarch 5/5 2023-01-25T05:41:08.970 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-1963.g61996d73.el8.noarch 1/5 2023-01-25T05:41:08.970 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-01-25T05:41:08.970 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-01-25T05:41:08.971 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-01-25T05:41:09.466 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-01-25T05:41:09.466 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:09.467 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2023-01-25T05:41:09.467 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm-2:18.0.0-1963.g61996d73.el8.noarch 2023-01-25T05:41:09.467 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-01-25T05:41:09.467 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel-2.5.1-7.el8.noarch 2023-01-25T05:41:09.467 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-01-25T05:41:09.468 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-01-25T05:41:09.468 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:09.468 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:41:09.577 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-fuse 2023-01-25T05:41:10.320 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:56 ago on Wed 25 Jan 2023 05:39:14 AM UTC. 2023-01-25T05:41:11.464 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:41:11.465 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:41:11.465 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2023-01-25T05:41:11.466 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:41:11.466 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2023-01-25T05:41:11.466 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse x86_64 2:18.0.0-1963.g61996d73.el8 ceph 845 k 2023-01-25T05:41:11.466 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2023-01-25T05:41:11.466 INFO:teuthology.orchestra.run.smithi203.stdout: fuse x86_64 2.9.7-16.el8 CentOS-BaseOS 83 k 2023-01-25T05:41:11.467 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:11.467 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2023-01-25T05:41:11.467 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:41:11.468 INFO:teuthology.orchestra.run.smithi203.stdout:Install 2 Packages 2023-01-25T05:41:11.468 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:11.469 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 928 k 2023-01-25T05:41:11.470 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 2.9 M 2023-01-25T05:41:11.470 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2023-01-25T05:41:11.533 INFO:teuthology.orchestra.run.smithi203.stdout:(1/2): fuse-2.9.7-16.el8.x86_64.rpm 1.2 MB/s | 83 kB 00:00 2023-01-25T05:41:11.967 INFO:teuthology.orchestra.run.smithi203.stdout:(2/2): ceph-fuse-18.0.0-1963.g61996d73.el8.x86_ 1.7 MB/s | 845 kB 00:00 2023-01-25T05:41:11.968 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2023-01-25T05:41:11.968 INFO:teuthology.orchestra.run.smithi203.stdout:Total 1.8 MB/s | 928 kB 00:00 2023-01-25T05:41:11.978 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2023-01-25T05:41:11.999 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2023-01-25T05:41:11.999 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2023-01-25T05:41:12.092 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2023-01-25T05:41:12.096 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2023-01-25T05:41:12.279 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2023-01-25T05:41:12.542 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : fuse-2.9.7-16.el8.x86_64 1/2 2023-01-25T05:41:12.792 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-fuse-2:18.0.0-1963.g61996d73.el8.x86_64 2/2 2023-01-25T05:41:13.191 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-fuse-2:18.0.0-1963.g61996d73.el8.x86_64 2/2 2023-01-25T05:41:13.192 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-fuse-2:18.0.0-1963.g61996d73.el8.x86_64 1/2 2023-01-25T05:41:13.898 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : fuse-2.9.7-16.el8.x86_64 2/2 2023-01-25T05:41:13.898 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:13.898 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2023-01-25T05:41:13.898 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-2:18.0.0-1963.g61996d73.el8.x86_64 fuse-2.9.7-16.el8.x86_64 2023-01-25T05:41:13.899 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:13.899 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:41:14.084 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-volume 2023-01-25T05:41:14.833 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:00 ago on Wed 25 Jan 2023 05:39:14 AM UTC. 2023-01-25T05:41:15.973 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:41:15.974 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:41:15.975 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2023-01-25T05:41:15.975 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:41:15.975 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2023-01-25T05:41:15.975 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume noarch 2:18.0.0-1963.g61996d73.el8 ceph-noarch 263 k 2023-01-25T05:41:15.975 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:15.976 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2023-01-25T05:41:15.976 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:41:15.976 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2023-01-25T05:41:15.976 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:15.977 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 263 k 2023-01-25T05:41:15.977 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 1.3 M 2023-01-25T05:41:15.977 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2023-01-25T05:41:16.296 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-volume-18.0.0-1963.g61996d73.el8.noarch.rp 822 kB/s | 263 kB 00:00 2023-01-25T05:41:16.297 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2023-01-25T05:41:16.297 INFO:teuthology.orchestra.run.smithi203.stdout:Total 817 kB/s | 263 kB 00:00 2023-01-25T05:41:16.297 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2023-01-25T05:41:16.307 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2023-01-25T05:41:16.307 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2023-01-25T05:41:16.408 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2023-01-25T05:41:16.411 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2023-01-25T05:41:16.598 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2023-01-25T05:41:16.835 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-volume-2:18.0.0-1963.g61996d73.el8.noarch 1/1 2023-01-25T05:41:17.272 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-volume-2:18.0.0-1963.g61996d73.el8.noarch 1/1 2023-01-25T05:41:17.736 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-volume-2:18.0.0-1963.g61996d73.el8.noarch 1/1 2023-01-25T05:41:17.737 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:17.737 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2023-01-25T05:41:17.737 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume-2:18.0.0-1963.g61996d73.el8.noarch 2023-01-25T05:41:17.737 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:17.737 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:41:17.927 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados-devel 2023-01-25T05:41:18.659 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:04 ago on Wed 25 Jan 2023 05:39:14 AM UTC. 2023-01-25T05:41:19.804 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:41:19.805 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:41:19.806 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2023-01-25T05:41:19.806 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:41:19.806 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2023-01-25T05:41:19.806 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel x86_64 2:18.0.0-1963.g61996d73.el8 ceph 139 k 2023-01-25T05:41:19.806 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:19.807 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2023-01-25T05:41:19.807 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:41:19.807 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2023-01-25T05:41:19.807 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:19.807 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 139 k 2023-01-25T05:41:19.808 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 512 k 2023-01-25T05:41:19.808 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2023-01-25T05:41:20.017 INFO:teuthology.orchestra.run.smithi203.stdout:librados-devel-18.0.0-1963.g61996d73.el8.x86_64 661 kB/s | 139 kB 00:00 2023-01-25T05:41:20.018 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2023-01-25T05:41:20.018 INFO:teuthology.orchestra.run.smithi203.stdout:Total 657 kB/s | 139 kB 00:00 2023-01-25T05:41:20.018 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2023-01-25T05:41:20.023 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2023-01-25T05:41:20.023 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2023-01-25T05:41:20.075 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2023-01-25T05:41:20.079 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2023-01-25T05:41:20.190 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2023-01-25T05:41:20.390 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados-devel-2:18.0.0-1963.g61996d73.el8.x86_64 1/1 2023-01-25T05:41:20.695 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados-devel-2:18.0.0-1963.g61996d73.el8.x86_64 1/1 2023-01-25T05:41:21.222 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados-devel-2:18.0.0-1963.g61996d73.el8.x86_64 1/1 2023-01-25T05:41:21.222 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:21.222 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2023-01-25T05:41:21.222 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:41:21.223 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:21.223 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:41:21.436 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install libcephfs2 2023-01-25T05:41:22.164 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:08 ago on Wed 25 Jan 2023 05:39:14 AM UTC. 2023-01-25T05:41:23.207 INFO:teuthology.orchestra.run.smithi203.stdout:Package libcephfs2-2:18.0.0-1963.g61996d73.el8.x86_64 is already installed. 2023-01-25T05:41:23.308 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:41:23.309 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2023-01-25T05:41:23.309 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:41:23.407 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install libcephfs-devel 2023-01-25T05:41:24.134 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:10 ago on Wed 25 Jan 2023 05:39:14 AM UTC. 2023-01-25T05:41:25.271 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:41:25.272 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:41:25.272 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2023-01-25T05:41:25.272 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:41:25.273 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2023-01-25T05:41:25.273 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel x86_64 2:18.0.0-1963.g61996d73.el8 ceph 30 k 2023-01-25T05:41:25.273 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:25.273 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2023-01-25T05:41:25.273 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:41:25.274 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2023-01-25T05:41:25.274 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:25.274 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 30 k 2023-01-25T05:41:25.274 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 137 k 2023-01-25T05:41:25.274 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2023-01-25T05:41:25.698 INFO:teuthology.orchestra.run.smithi203.stdout:libcephfs-devel-18.0.0-1963.g61996d73.el8.x86_6 72 kB/s | 30 kB 00:00 2023-01-25T05:41:25.699 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2023-01-25T05:41:25.699 INFO:teuthology.orchestra.run.smithi203.stdout:Total 71 kB/s | 30 kB 00:00 2023-01-25T05:41:25.699 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2023-01-25T05:41:25.702 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2023-01-25T05:41:25.702 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2023-01-25T05:41:25.731 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2023-01-25T05:41:25.734 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2023-01-25T05:41:25.794 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2023-01-25T05:41:25.989 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs-devel-2:18.0.0-1963.g61996d73.el8.x86_64 1/1 2023-01-25T05:41:26.282 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-1963.g61996d73.el8.x86_64 1/1 2023-01-25T05:41:26.730 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs-devel-2:18.0.0-1963.g61996d73.el8.x86_64 1/1 2023-01-25T05:41:26.731 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:26.731 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2023-01-25T05:41:26.731 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:41:26.731 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:26.731 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:41:26.833 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados2 2023-01-25T05:41:27.566 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:13 ago on Wed 25 Jan 2023 05:39:14 AM UTC. 2023-01-25T05:41:28.600 INFO:teuthology.orchestra.run.smithi203.stdout:Package librados2-2:18.0.0-1963.g61996d73.el8.x86_64 is already installed. 2023-01-25T05:41:28.701 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:41:28.702 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2023-01-25T05:41:28.702 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:41:28.792 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librbd1 2023-01-25T05:41:29.527 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:15 ago on Wed 25 Jan 2023 05:39:14 AM UTC. 2023-01-25T05:41:30.568 INFO:teuthology.orchestra.run.smithi203.stdout:Package librbd1-2:18.0.0-1963.g61996d73.el8.x86_64 is already installed. 2023-01-25T05:41:30.670 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:41:30.671 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2023-01-25T05:41:30.671 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:41:30.767 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rados 2023-01-25T05:41:31.504 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:17 ago on Wed 25 Jan 2023 05:39:14 AM UTC. 2023-01-25T05:41:32.542 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rados-2:18.0.0-1963.g61996d73.el8.x86_64 is already installed. 2023-01-25T05:41:32.642 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:41:32.643 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2023-01-25T05:41:32.644 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:41:32.751 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rgw 2023-01-25T05:41:33.483 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:19 ago on Wed 25 Jan 2023 05:39:14 AM UTC. 2023-01-25T05:41:34.527 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rgw-2:18.0.0-1963.g61996d73.el8.x86_64 is already installed. 2023-01-25T05:41:34.629 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:41:34.630 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2023-01-25T05:41:34.630 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:41:34.722 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-cephfs 2023-01-25T05:41:35.445 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:21 ago on Wed 25 Jan 2023 05:39:14 AM UTC. 2023-01-25T05:41:36.501 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-cephfs-2:18.0.0-1963.g61996d73.el8.x86_64 is already installed. 2023-01-25T05:41:36.602 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:41:36.603 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2023-01-25T05:41:36.603 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:41:36.701 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rbd 2023-01-25T05:41:37.423 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:23 ago on Wed 25 Jan 2023 05:39:14 AM UTC. 2023-01-25T05:41:38.468 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rbd-2:18.0.0-1963.g61996d73.el8.x86_64 is already installed. 2023-01-25T05:41:38.565 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:41:38.566 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2023-01-25T05:41:38.566 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:41:38.658 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-fuse 2023-01-25T05:41:39.393 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:25 ago on Wed 25 Jan 2023 05:39:14 AM UTC. 2023-01-25T05:41:40.537 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:41:40.538 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:41:40.538 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2023-01-25T05:41:40.538 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:41:40.538 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2023-01-25T05:41:40.539 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse x86_64 2:18.0.0-1963.g61996d73.el8 ceph 94 k 2023-01-25T05:41:40.539 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:40.539 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2023-01-25T05:41:40.539 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:41:40.539 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2023-01-25T05:41:40.540 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:40.540 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 94 k 2023-01-25T05:41:40.540 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 280 k 2023-01-25T05:41:40.540 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2023-01-25T05:41:40.746 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-fuse-18.0.0-1963.g61996d73.el8.x86_64.rpm 455 kB/s | 94 kB 00:00 2023-01-25T05:41:40.747 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2023-01-25T05:41:40.747 INFO:teuthology.orchestra.run.smithi203.stdout:Total 452 kB/s | 94 kB 00:00 2023-01-25T05:41:40.747 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2023-01-25T05:41:40.757 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2023-01-25T05:41:40.757 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2023-01-25T05:41:40.810 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2023-01-25T05:41:40.813 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2023-01-25T05:41:41.341 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2023-01-25T05:41:41.628 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-fuse-2:18.0.0-1963.g61996d73.el8.x86_64 1/1 2023-01-25T05:41:41.916 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-fuse-2:18.0.0-1963.g61996d73.el8.x86_64 1/1 2023-01-25T05:41:42.445 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-fuse-2:18.0.0-1963.g61996d73.el8.x86_64 1/1 2023-01-25T05:41:42.446 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:42.446 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2023-01-25T05:41:42.446 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:41:42.446 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:42.447 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:41:42.638 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-mirror 2023-01-25T05:41:43.369 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:29 ago on Wed 25 Jan 2023 05:39:14 AM UTC. 2023-01-25T05:41:44.509 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:41:44.509 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:41:44.510 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2023-01-25T05:41:44.510 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:41:44.510 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2023-01-25T05:41:44.510 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror x86_64 2:18.0.0-1963.g61996d73.el8 ceph 3.4 M 2023-01-25T05:41:44.510 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:44.510 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2023-01-25T05:41:44.511 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:41:44.511 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2023-01-25T05:41:44.511 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:44.512 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 3.4 M 2023-01-25T05:41:44.512 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 16 M 2023-01-25T05:41:44.512 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2023-01-25T05:41:44.986 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-mirror-18.0.0-1963.g61996d73.el8.x86_64.rpm 7.3 MB/s | 3.4 MB 00:00 2023-01-25T05:41:44.987 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2023-01-25T05:41:44.987 INFO:teuthology.orchestra.run.smithi203.stdout:Total 7.2 MB/s | 3.4 MB 00:00 2023-01-25T05:41:44.988 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2023-01-25T05:41:44.999 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2023-01-25T05:41:45.000 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2023-01-25T05:41:45.062 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2023-01-25T05:41:45.066 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2023-01-25T05:41:45.503 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2023-01-25T05:41:45.672 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-mirror-2:18.0.0-1963.g61996d73.el8.x86_64 1/1 2023-01-25T05:41:46.117 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-mirror-2:18.0.0-1963.g61996d73.el8.x86_64 1/1 2023-01-25T05:41:46.680 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-mirror-2:18.0.0-1963.g61996d73.el8.x86_64 1/1 2023-01-25T05:41:46.680 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:46.680 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2023-01-25T05:41:46.680 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:41:46.681 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:46.681 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:41:46.864 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-nbd 2023-01-25T05:41:47.614 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:33 ago on Wed 25 Jan 2023 05:39:14 AM UTC. 2023-01-25T05:41:48.763 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-01-25T05:41:48.763 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:41:48.764 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2023-01-25T05:41:48.764 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:41:48.764 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2023-01-25T05:41:48.764 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd x86_64 2:18.0.0-1963.g61996d73.el8 ceph 179 k 2023-01-25T05:41:48.764 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:48.765 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2023-01-25T05:41:48.765 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-01-25T05:41:48.765 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2023-01-25T05:41:48.765 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:48.765 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 179 k 2023-01-25T05:41:48.766 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 568 k 2023-01-25T05:41:48.766 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2023-01-25T05:41:48.958 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-nbd-18.0.0-1963.g61996d73.el8.x86_64.rpm 924 kB/s | 179 kB 00:00 2023-01-25T05:41:48.959 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2023-01-25T05:41:48.959 INFO:teuthology.orchestra.run.smithi203.stdout:Total 918 kB/s | 179 kB 00:00 2023-01-25T05:41:48.959 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2023-01-25T05:41:48.970 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2023-01-25T05:41:48.970 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2023-01-25T05:41:49.022 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2023-01-25T05:41:49.025 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2023-01-25T05:41:49.152 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2023-01-25T05:41:49.326 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-nbd-2:18.0.0-1963.g61996d73.el8.x86_64 1/1 2023-01-25T05:41:49.645 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-nbd-2:18.0.0-1963.g61996d73.el8.x86_64 1/1 2023-01-25T05:41:50.165 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-nbd-2:18.0.0-1963.g61996d73.el8.x86_64 1/1 2023-01-25T05:41:50.165 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:50.165 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2023-01-25T05:41:50.166 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-2:18.0.0-1963.g61996d73.el8.x86_64 2023-01-25T05:41:50.166 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:50.166 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-01-25T05:41:50.354 DEBUG:teuthology.parallel:result is None 2023-01-25T05:41:50.355 DEBUG:teuthology.orchestra.run.smithi203:> sudo lsb_release -is 2023-01-25T05:41:50.393 INFO:teuthology.orchestra.run.smithi203.stdout:CentOSStream 2023-01-25T05:41:50.394 DEBUG:teuthology.misc:System to be installed: CentOSStream 2023-01-25T05:41:50.395 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=61996d73ab738798e4ee0055c86457f5c459da4a 2023-01-25T05:41:50.544 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-01-25T05:41:50.567 INFO:teuthology.orchestra.run.smithi203.stdout:18.0.0-1963.g61996d73.el8 2023-01-25T05:41:50.568 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-1963.g61996d73.el8 2023-01-25T05:41:50.568 INFO:teuthology.task.install:The correct ceph version 18.0.0-1963.g61996d73 is installed. 2023-01-25T05:41:50.568 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-01-25T05:41:50.569 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:41:50.569 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-01-25T05:41:50.634 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-01-25T05:41:50.635 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:41:50.635 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/daemon-helper 2023-01-25T05:41:50.702 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-01-25T05:41:50.769 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-01-25T05:41:50.770 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:41:50.770 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/adjust-ulimits 2023-01-25T05:41:50.835 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-01-25T05:41:50.903 INFO:teuthology.run_tasks:Running task ceph... 2023-01-25T05:41:50.963 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-01-25T05:41:50.964 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 777 /var/log/ceph 2023-01-25T05:41:50.991 INFO:tasks.ceph:Disabling ceph logrotate... 2023-01-25T05:41:50.992 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/logrotate.d/ceph 2023-01-25T05:41:51.059 INFO:tasks.ceph:Creating extra log directories... 2023-01-25T05:41:51.059 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-01-25T05:41:51.135 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-01-25T05:41:51.135 INFO:tasks.ceph:config {'conf': {'osd': {'osd max object name len': 400, 'osd max object namespace len': 64, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-01-25T05:41:51.136 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-01-24_22:22:09-rados-wip-yuri2-testing-2023-01-23-0928-distro-default-smithi/7136839', 'branch': 'wip-yuri2-testing-2023-01-23-0928', 'description': 'rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_8}}', 'email': 'yweinstein@radhat.com', 'first_in_suite': False, 'job_id': '7136839', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-01-24_22:22:09-rados-wip-yuri2-testing-2023-01-23-0928-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri2-testing-2023-01-23-0928'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '61996d73ab738798e4ee0055c86457f5c459da4a'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '61996d73ab738798e4ee0055c86457f5c459da4a'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-yuri2-testing-2023-01-23-0928', 'sha1': '61996d73ab738798e4ee0055c86457f5c459da4a'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 71, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'osd.8', 'osd.9', 'client.0']], 'sha1': '61996d73ab738798e4ee0055c86457f5c459da4a', 'sleep_before_teardown': 0, 'suite': 'rados', 'suite_branch': 'wip-yuri2-testing-2023-01-23-0928', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_61996d73ab738798e4ee0055c86457f5c459da4a/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '61996d73ab738798e4ee0055c86457f5c459da4a', 'targets': {'smithi203.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC55Zz3HM1j0J7oxuTWSJhvf5bmHHheR3iKfuBOghJmm13VaucE0tUh59xKnzSpKUY8McQvELXBroAXrLBmaPNqu44vmbptfI0EYPgZYkDELfwxxwCE6w9GPqARlnNlG9qTLb/ZkCke4Scfdv2BS52z14nlgntP43O3Yprvqoi2FRD5x7GmS769OiEpWg4N2UYFiyPNk5aumG6wbJfW7MUg414oKllbBScP4H4JodAlMmLgj1KolkzGqq2BdcPCJk3qilnuyzde46cPKWrorCFCVlZ/TvG/5r+lCIazNSBBf43qY/k+oG+M9EHFGEBv722w3X1cpreV4k7Q7CMa1X7q3SZ0InX9GurHNMcFkxu+Z1M9kzHFBi8aPM6SLtyk5HMd9em2xwj67Gt3WdSJcysteKg9D2VaZMKDsAEcw76zsr/URN5fuXCeotNy25dOpALkVREQxel0KPMaoID17pgUOuxqlTcb7xyuCoODb7lfcn2h0kR3JBTHViTK0wQVaG8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'osd max object name len': 400, 'osd max object namespace len': 64, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'log-ignorelist': ['but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'flavor': 'default', 'sha1': '61996d73ab738798e4ee0055c86457f5c459da4a', 'cluster': 'ceph'}}, {'workunit': {'clients': {'all': ['rados/test_health_warnings.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '8e1bba12e584b7ae912e92e94afe9c075ab51fa3', 'timestamp': '2023-01-24_22:22:09', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.17735'} 2023-01-25T05:41:51.136 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-01-25T05:41:51.185 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/run/ceph 2023-01-25T05:41:51.254 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:41:51.254 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/scratch_devs of=/dev/stdout 2023-01-25T05:41:51.312 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-01-25T05:41:51.312 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_1 2023-01-25T05:41:51.370 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-01-25T05:41:51.370 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-25T05:41:51.370 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 6h/6d Inode: 373874 Links: 1 2023-01-25T05:41:51.371 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-25T05:41:51.371 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2023-01-25T05:41:51.371 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2023-01-25 05:41:45.850395071 +0000 2023-01-25T05:41:51.371 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2023-01-25 05:38:27.717308655 +0000 2023-01-25T05:41:51.371 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2023-01-25 05:38:27.717308655 +0000 2023-01-25T05:41:51.372 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2023-01-25T05:41:51.372 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-01-25T05:41:51.436 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2023-01-25T05:41:51.436 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2023-01-25T05:41:51.436 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000215515 s, 2.4 MB/s 2023-01-25T05:41:51.437 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-01-25T05:41:51.497 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_2 2023-01-25T05:41:51.555 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-01-25T05:41:51.555 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-25T05:41:51.555 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 6h/6d Inode: 370531 Links: 1 2023-01-25T05:41:51.556 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-25T05:41:51.556 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2023-01-25T05:41:51.556 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2023-01-25 05:41:45.850395071 +0000 2023-01-25T05:41:51.556 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2023-01-25 05:38:28.382292525 +0000 2023-01-25T05:41:51.557 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2023-01-25 05:38:28.382292525 +0000 2023-01-25T05:41:51.557 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2023-01-25T05:41:51.557 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-01-25T05:41:51.621 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2023-01-25T05:41:51.621 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2023-01-25T05:41:51.621 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000185295 s, 2.8 MB/s 2023-01-25T05:41:51.622 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-01-25T05:41:51.680 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_3 2023-01-25T05:41:51.738 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-01-25T05:41:51.739 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-25T05:41:51.739 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 6h/6d Inode: 373669 Links: 1 2023-01-25T05:41:51.739 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-25T05:41:51.739 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2023-01-25T05:41:51.740 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2023-01-25 05:41:45.850395071 +0000 2023-01-25T05:41:51.740 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2023-01-25 05:38:28.905279840 +0000 2023-01-25T05:41:51.740 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2023-01-25 05:38:28.905279840 +0000 2023-01-25T05:41:51.740 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2023-01-25T05:41:51.741 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-01-25T05:41:51.805 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2023-01-25T05:41:51.805 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2023-01-25T05:41:51.805 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000280144 s, 1.8 MB/s 2023-01-25T05:41:51.806 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-01-25T05:41:51.866 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_4 2023-01-25T05:41:51.924 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-01-25T05:41:51.924 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-25T05:41:51.924 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 6h/6d Inode: 376087 Links: 1 2023-01-25T05:41:51.924 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-25T05:41:51.925 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2023-01-25T05:41:51.925 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2023-01-25 05:41:45.850395071 +0000 2023-01-25T05:41:51.925 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2023-01-25 05:38:29.416267446 +0000 2023-01-25T05:41:51.925 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2023-01-25 05:38:29.416267446 +0000 2023-01-25T05:41:51.925 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: - 2023-01-25T05:41:51.926 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-01-25T05:41:51.990 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2023-01-25T05:41:51.990 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2023-01-25T05:41:51.990 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000166722 s, 3.1 MB/s 2023-01-25T05:41:51.991 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-01-25T05:41:52.050 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-25T05:41:52.051 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2023-01-25T05:41:52.051 INFO:tasks.ceph:Generating config... 2023-01-25T05:41:52.053 INFO:tasks.ceph:[osd] osd max object name len = 400 2023-01-25T05:41:52.053 INFO:tasks.ceph:[osd] osd max object namespace len = 64 2023-01-25T05:41:52.053 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-01-25T05:41:52.053 INFO:tasks.ceph:[osd] debug ms = 1 2023-01-25T05:41:52.054 INFO:tasks.ceph:[osd] debug osd = 20 2023-01-25T05:41:52.054 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-01-25T05:41:52.054 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-01-25T05:41:52.055 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-01-25T05:41:52.055 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-01-25T05:41:52.055 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-01-25T05:41:52.055 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-01-25T05:41:52.056 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-01-25T05:41:52.056 INFO:tasks.ceph:[mgr] debug ms = 1 2023-01-25T05:41:52.057 INFO:tasks.ceph:[mon] debug mon = 20 2023-01-25T05:41:52.058 INFO:tasks.ceph:[mon] debug ms = 1 2023-01-25T05:41:52.058 INFO:tasks.ceph:[mon] debug paxos = 20 2023-01-25T05:41:52.058 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-01-25T05:41:52.058 INFO:tasks.ceph:Setting up mon.a... 2023-01-25T05:41:52.059 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-01-25T05:41:52.146 INFO:teuthology.orchestra.run.smithi203.stdout:creating /etc/ceph/ceph.keyring 2023-01-25T05:41:52.149 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-01-25T05:41:52.194 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-25T05:41:52.221 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.203')] 2023-01-25T05:41:52.222 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.203', 'mon election default strategy': 1}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'osd max object name len': 400, 'osd max object namespace len': 64, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd mclock override recovery settings': True, 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-01-25T05:41:52.223 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:41:52.223 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-01-25T05:41:52.280 DEBUG:teuthology.orchestra.run.smithi203:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.203 --print /home/ubuntu/cephtest/ceph.monmap 2023-01-25T05:41:52.362 INFO:teuthology.orchestra.run.smithi203.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-01-25T05:41:52.363 INFO:teuthology.orchestra.run.smithi203.stdout:monmaptool: generated fsid 5b00363d-2783-4934-a70f-2b1720d57fe5 2023-01-25T05:41:52.363 INFO:teuthology.orchestra.run.smithi203.stdout:setting min_mon_release = pacific 2023-01-25T05:41:52.363 INFO:teuthology.orchestra.run.smithi203.stdout:epoch 0 2023-01-25T05:41:52.363 INFO:teuthology.orchestra.run.smithi203.stdout:fsid 5b00363d-2783-4934-a70f-2b1720d57fe5 2023-01-25T05:41:52.363 INFO:teuthology.orchestra.run.smithi203.stdout:last_changed 2023-01-25T05:41:52.362056+0000 2023-01-25T05:41:52.364 INFO:teuthology.orchestra.run.smithi203.stdout:created 2023-01-25T05:41:52.362056+0000 2023-01-25T05:41:52.364 INFO:teuthology.orchestra.run.smithi203.stdout:min_mon_release 16 (pacific) 2023-01-25T05:41:52.364 INFO:teuthology.orchestra.run.smithi203.stdout:election_strategy: 1 2023-01-25T05:41:52.364 INFO:teuthology.orchestra.run.smithi203.stdout:0: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.a 2023-01-25T05:41:52.364 INFO:teuthology.orchestra.run.smithi203.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-01-25T05:41:52.366 DEBUG:teuthology.orchestra.run.smithi203:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-01-25T05:41:52.423 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 5b00363d-2783-4934-a70f-2b1720d57fe5... 2023-01-25T05:41:52.425 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-01-25T05:41:52.509 INFO:teuthology.orchestra.run.smithi203.stdout:[global] 2023-01-25T05:41:52.509 INFO:teuthology.orchestra.run.smithi203.stdout: chdir = "" 2023-01-25T05:41:52.510 INFO:teuthology.orchestra.run.smithi203.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-01-25T05:41:52.510 INFO:teuthology.orchestra.run.smithi203.stdout: auth supported = cephx 2023-01-25T05:41:52.510 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.510 INFO:teuthology.orchestra.run.smithi203.stdout: filestore xattr use omap = true 2023-01-25T05:41:52.511 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.511 INFO:teuthology.orchestra.run.smithi203.stdout: mon clock drift allowed = 1.000 2023-01-25T05:41:52.511 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.511 INFO:teuthology.orchestra.run.smithi203.stdout: osd crush chooseleaf type = 0 2023-01-25T05:41:52.512 INFO:teuthology.orchestra.run.smithi203.stdout: auth debug = true 2023-01-25T05:41:52.512 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.512 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on old message = true 2023-01-25T05:41:52.512 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on bug = true 2023-01-25T05:41:52.513 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.513 INFO:teuthology.orchestra.run.smithi203.stdout: mon max pg per osd = 10000 # >= luminous 2023-01-25T05:41:52.513 INFO:teuthology.orchestra.run.smithi203.stdout: mon pg warn max object skew = 0 2023-01-25T05:41:52.513 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.513 INFO:teuthology.orchestra.run.smithi203.stdout: # disable pg_autoscaler by default for new pools 2023-01-25T05:41:52.513 INFO:teuthology.orchestra.run.smithi203.stdout: osd_pool_default_pg_autoscale_mode = off 2023-01-25T05:41:52.514 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.514 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default size = 2 2023-01-25T05:41:52.514 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.514 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow primary affinity = true 2023-01-25T05:41:52.514 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow pg remap = true 2023-01-25T05:41:52.515 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on legacy crush tunables = false 2023-01-25T05:41:52.515 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on crush straw calc version zero = false 2023-01-25T05:41:52.515 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on no sortbitwise = false 2023-01-25T05:41:52.515 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on osd down out interval zero = false 2023-01-25T05:41:52.515 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on too few osds = false 2023-01-25T05:41:52.515 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-01-25T05:41:52.516 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_no_redundancy = false 2023-01-25T05:41:52.516 INFO:teuthology.orchestra.run.smithi203.stdout: mon_allow_pool_size_one = true 2023-01-25T05:41:52.516 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.516 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-01-25T05:41:52.516 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.517 INFO:teuthology.orchestra.run.smithi203.stdout: osd default data pool replay window = 5 2023-01-25T05:41:52.517 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.517 INFO:teuthology.orchestra.run.smithi203.stdout: mon allow pool delete = true 2023-01-25T05:41:52.517 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.517 INFO:teuthology.orchestra.run.smithi203.stdout: mon cluster log file level = debug 2023-01-25T05:41:52.518 INFO:teuthology.orchestra.run.smithi203.stdout: debug asserts on shutdown = true 2023-01-25T05:41:52.518 INFO:teuthology.orchestra.run.smithi203.stdout: mon health detail to clog = false 2023-01-25T05:41:52.518 INFO:teuthology.orchestra.run.smithi203.stdout: mon host = 172.21.15.203 2023-01-25T05:41:52.518 INFO:teuthology.orchestra.run.smithi203.stdout: mon election default strategy = 1 2023-01-25T05:41:52.518 INFO:teuthology.orchestra.run.smithi203.stdout: fsid = 5b00363d-2783-4934-a70f-2b1720d57fe5 2023-01-25T05:41:52.518 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.519 INFO:teuthology.orchestra.run.smithi203.stdout:[osd] 2023-01-25T05:41:52.519 INFO:teuthology.orchestra.run.smithi203.stdout: osd journal size = 100 2023-01-25T05:41:52.519 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.519 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub load threshold = 5.0 2023-01-25T05:41:52.519 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub max interval = 600 2023-01-25T05:41:52.520 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.520 INFO:teuthology.orchestra.run.smithi203.stdout: osd recover clone overlap = true 2023-01-25T05:41:52.520 INFO:teuthology.orchestra.run.smithi203.stdout: osd recovery max chunk = 1048576 2023-01-25T05:41:52.520 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.520 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug shutdown = true 2023-01-25T05:41:52.521 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug op order = true 2023-01-25T05:41:52.521 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify stray on activate = true 2023-01-25T05:41:52.521 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.521 INFO:teuthology.orchestra.run.smithi203.stdout: osd open classes on start = true 2023-01-25T05:41:52.522 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug pg log writeout = true 2023-01-25T05:41:52.522 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.522 INFO:teuthology.orchestra.run.smithi203.stdout: osd deep scrub update digest min age = 30 2023-01-25T05:41:52.522 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.522 INFO:teuthology.orchestra.run.smithi203.stdout: osd map max advance = 10 2023-01-25T05:41:52.523 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.523 INFO:teuthology.orchestra.run.smithi203.stdout: journal zero on create = true 2023-01-25T05:41:52.523 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.523 INFO:teuthology.orchestra.run.smithi203.stdout: filestore ondisk finisher threads = 3 2023-01-25T05:41:52.523 INFO:teuthology.orchestra.run.smithi203.stdout: filestore apply finisher threads = 3 2023-01-25T05:41:52.524 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.524 INFO:teuthology.orchestra.run.smithi203.stdout: bdev debug aio = true 2023-01-25T05:41:52.524 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug misdirected ops = true 2023-01-25T05:41:52.524 INFO:teuthology.orchestra.run.smithi203.stdout: osd max object name len = 400 2023-01-25T05:41:52.524 INFO:teuthology.orchestra.run.smithi203.stdout: osd max object namespace len = 64 2023-01-25T05:41:52.525 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore zero block detection = True 2023-01-25T05:41:52.525 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2023-01-25T05:41:52.525 INFO:teuthology.orchestra.run.smithi203.stdout: debug osd = 20 2023-01-25T05:41:52.525 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify cached snaps = True 2023-01-25T05:41:52.525 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify missing on start = True 2023-01-25T05:41:52.526 INFO:teuthology.orchestra.run.smithi203.stdout: osd mclock override recovery settings = True 2023-01-25T05:41:52.526 INFO:teuthology.orchestra.run.smithi203.stdout: osd op queue = debug_random 2023-01-25T05:41:52.526 INFO:teuthology.orchestra.run.smithi203.stdout: osd op queue cut off = debug_random 2023-01-25T05:41:52.526 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.526 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr] 2023-01-25T05:41:52.527 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2023-01-25T05:41:52.527 INFO:teuthology.orchestra.run.smithi203.stdout: debug mgr = 20 2023-01-25T05:41:52.527 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2023-01-25T05:41:52.527 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2023-01-25T05:41:52.528 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2023-01-25T05:41:52.528 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2023-01-25T05:41:52.528 INFO:teuthology.orchestra.run.smithi203.stdout: mgr/telemetry/nag = false 2023-01-25T05:41:52.528 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.528 INFO:teuthology.orchestra.run.smithi203.stdout:[mon] 2023-01-25T05:41:52.528 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2023-01-25T05:41:52.529 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2023-01-25T05:41:52.529 INFO:teuthology.orchestra.run.smithi203.stdout: debug paxos = 20 2023-01-25T05:41:52.529 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2023-01-25T05:41:52.529 INFO:teuthology.orchestra.run.smithi203.stdout: mon data avail warn = 5 2023-01-25T05:41:52.529 INFO:teuthology.orchestra.run.smithi203.stdout: mon mgr mkfs grace = 240 2023-01-25T05:41:52.530 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2023-01-25T05:41:52.530 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd reporter subtree level = osd 2023-01-25T05:41:52.530 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd prime pg temp = true 2023-01-25T05:41:52.530 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2023-01-25T05:41:52.530 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.531 INFO:teuthology.orchestra.run.smithi203.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-01-25T05:41:52.531 INFO:teuthology.orchestra.run.smithi203.stdout: auth mon ticket ttl = 660 # 11m 2023-01-25T05:41:52.531 INFO:teuthology.orchestra.run.smithi203.stdout: auth service ticket ttl = 240 # 4m 2023-01-25T05:41:52.531 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.531 INFO:teuthology.orchestra.run.smithi203.stdout: # don't complain about insecure global_id in the test suite 2023-01-25T05:41:52.532 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-01-25T05:41:52.532 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-01-25T05:41:52.532 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.532 INFO:teuthology.orchestra.run.smithi203.stdout: # 1m isn't quite enough 2023-01-25T05:41:52.532 INFO:teuthology.orchestra.run.smithi203.stdout: mon_down_mkfs_grace = 2m 2023-01-25T05:41:52.533 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.533 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_filestore_osds = false 2023-01-25T05:41:52.533 INFO:teuthology.orchestra.run.smithi203.stdout: mon scrub interval = 300 2023-01-25T05:41:52.533 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:41:52.533 INFO:teuthology.orchestra.run.smithi203.stdout:[client] 2023-01-25T05:41:52.534 INFO:teuthology.orchestra.run.smithi203.stdout: rgw cache enabled = true 2023-01-25T05:41:52.534 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable ops log = true 2023-01-25T05:41:52.534 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable usage log = true 2023-01-25T05:41:52.534 INFO:teuthology.orchestra.run.smithi203.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-01-25T05:41:52.534 INFO:teuthology.orchestra.run.smithi203.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-01-25T05:41:52.534 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.a] 2023-01-25T05:41:52.535 INFO:tasks.ceph:Creating admin key on mon.a... 2023-01-25T05:41:52.536 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-01-25T05:41:52.633 INFO:tasks.ceph:Copying monmap to all nodes... 2023-01-25T05:41:52.634 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:41:52.634 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-01-25T05:41:52.652 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:41:52.652 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-01-25T05:41:52.709 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi203.front.sepia.ceph.com 2023-01-25T05:41:52.710 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:41:52.710 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.keyring 2023-01-25T05:41:52.711 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-25T05:41:52.786 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:41:52.787 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-01-25T05:41:52.845 INFO:tasks.ceph:Setting up mon nodes... 2023-01-25T05:41:52.845 INFO:tasks.ceph:Setting up mgr nodes... 2023-01-25T05:41:52.846 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-01-25T05:41:52.939 INFO:teuthology.orchestra.run.smithi203.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-01-25T05:41:52.941 INFO:tasks.ceph:Setting up mds nodes... 2023-01-25T05:41:52.942 INFO:tasks.ceph_client:Setting up client nodes... 2023-01-25T05:41:52.946 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-01-25T05:41:52.990 INFO:teuthology.orchestra.run.smithi203.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-01-25T05:41:53.002 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-01-25T05:41:53.003 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2023-01-25T05:41:53.003 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-01-25T05:41:53.031 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-25T05:41:53.031 INFO:tasks.ceph:role: osd.0 2023-01-25T05:41:53.031 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com 2023-01-25T05:41:53.032 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-01-25T05:41:53.106 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-01-25T05:41:53.106 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-25T05:41:53.107 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-25T05:41:53.107 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-25T05:41:53.107 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-25T05:41:53.107 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2023-01-25T05:41:53.107 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-25T05:41:53.108 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-25T05:41:53.108 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-25T05:41:53.108 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-25T05:41:53.544 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2023-01-25T05:41:53.546 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2023-01-25T05:41:53.546 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-01-25T05:41:53.722 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-01-25T05:41:53.756 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-01-25T05:41:53.823 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-25T05:41:53.824 INFO:tasks.ceph:role: osd.1 2023-01-25T05:41:53.824 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com 2023-01-25T05:41:53.824 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-01-25T05:41:53.898 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-01-25T05:41:53.898 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-25T05:41:53.899 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-25T05:41:53.899 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-25T05:41:53.899 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-25T05:41:53.899 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2023-01-25T05:41:53.899 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-25T05:41:53.900 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-25T05:41:53.900 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-25T05:41:53.900 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-25T05:41:54.323 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2023-01-25T05:41:54.325 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2023-01-25T05:41:54.325 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-01-25T05:41:54.435 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-01-25T05:41:54.468 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-01-25T05:41:54.536 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-25T05:41:54.536 INFO:tasks.ceph:role: osd.2 2023-01-25T05:41:54.536 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi203.front.sepia.ceph.com 2023-01-25T05:41:54.537 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-01-25T05:41:54.611 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-01-25T05:41:54.611 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-25T05:41:54.611 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-25T05:41:54.611 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-25T05:41:54.612 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-25T05:41:54.612 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2023-01-25T05:41:54.612 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-25T05:41:54.612 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-25T05:41:54.612 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-25T05:41:54.613 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-25T05:41:55.054 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2023-01-25T05:41:55.055 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2023-01-25T05:41:55.056 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-01-25T05:41:55.205 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-01-25T05:41:55.238 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-01-25T05:41:55.307 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-25T05:41:55.307 INFO:tasks.ceph:role: osd.3 2023-01-25T05:41:55.307 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi203.front.sepia.ceph.com 2023-01-25T05:41:55.308 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-01-25T05:41:55.382 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-01-25T05:41:55.382 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-25T05:41:55.383 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-25T05:41:55.383 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-01-25T05:41:55.383 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-25T05:41:55.383 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2023-01-25T05:41:55.383 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-25T05:41:55.384 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-25T05:41:55.384 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-25T05:41:55.384 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-25T05:41:55.834 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2023-01-25T05:41:55.836 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2023-01-25T05:41:55.836 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-01-25T05:41:55.981 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-01-25T05:41:56.013 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-01-25T05:41:56.081 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-25T05:41:56.082 INFO:tasks.ceph:role: osd.4 2023-01-25T05:41:56.082 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-01-25T05:41:56.148 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-25T05:41:56.148 INFO:tasks.ceph:role: osd.5 2023-01-25T05:41:56.149 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-01-25T05:41:56.215 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-25T05:41:56.216 INFO:tasks.ceph:role: osd.6 2023-01-25T05:41:56.216 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-01-25T05:41:56.283 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-25T05:41:56.284 INFO:tasks.ceph:role: osd.7 2023-01-25T05:41:56.284 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2023-01-25T05:41:56.351 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-25T05:41:56.351 INFO:tasks.ceph:role: osd.8 2023-01-25T05:41:56.352 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2023-01-25T05:41:56.418 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-25T05:41:56.418 INFO:tasks.ceph:role: osd.9 2023-01-25T05:41:56.419 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-25T05:41:56.511 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:41:56.509+0000 7febb5e745c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-01-25T05:41:56.511 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:41:56.510+0000 7febb5e745c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-01-25T05:41:56.512 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:41:56.510+0000 7febb5e745c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-01-25T05:41:56.512 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:41:56.510+0000 7febb5e745c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-01-25T05:41:56.512 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:41:56.510+0000 7febb5e745c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-01-25T05:41:56.512 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:41:56.510+0000 7febb5e745c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-01-25T05:41:57.035 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:41:57.034+0000 7febb5e745c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T05:41:57.800 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:41:57.799+0000 7febb5e745c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T05:41:58.587 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-01-25T05:41:58.617 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-25T05:41:58.706 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:41:58.705+0000 7f3313ddc5c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-01-25T05:41:58.707 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:41:58.705+0000 7f3313ddc5c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-01-25T05:41:58.707 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:41:58.705+0000 7f3313ddc5c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-01-25T05:41:58.707 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:41:58.705+0000 7f3313ddc5c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-01-25T05:41:58.708 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:41:58.705+0000 7f3313ddc5c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-01-25T05:41:58.708 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:41:58.706+0000 7f3313ddc5c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-01-25T05:41:59.228 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:41:59.227+0000 7f3313ddc5c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T05:41:59.993 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:41:59.992+0000 7f3313ddc5c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T05:42:00.777 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-01-25T05:42:00.806 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-25T05:42:00.894 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:00.892+0000 7fe4e1c735c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-01-25T05:42:00.894 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:00.892+0000 7fe4e1c735c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-01-25T05:42:00.894 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:00.892+0000 7fe4e1c735c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-01-25T05:42:00.894 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:00.892+0000 7fe4e1c735c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-01-25T05:42:00.895 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:00.892+0000 7fe4e1c735c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-01-25T05:42:00.895 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:00.893+0000 7fe4e1c735c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-01-25T05:42:01.425 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:01.424+0000 7fe4e1c735c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T05:42:02.190 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:02.188+0000 7fe4e1c735c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T05:42:02.975 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-01-25T05:42:03.005 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-25T05:42:03.095 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:03.092+0000 7f50cd0df5c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-01-25T05:42:03.095 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:03.092+0000 7f50cd0df5c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-01-25T05:42:03.096 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:03.092+0000 7f50cd0df5c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-01-25T05:42:03.096 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:03.092+0000 7f50cd0df5c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-01-25T05:42:03.096 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:03.092+0000 7f50cd0df5c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-01-25T05:42:03.096 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:03.093+0000 7f50cd0df5c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-01-25T05:42:03.619 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:03.617+0000 7f50cd0df5c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T05:42:04.381 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:04.379+0000 7f50cd0df5c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T05:42:05.169 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-01-25T05:42:05.198 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-25T05:42:05.286 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:05.283+0000 7fe6c53395c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-01-25T05:42:05.286 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:05.283+0000 7fe6c53395c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-01-25T05:42:05.286 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:05.283+0000 7fe6c53395c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-01-25T05:42:05.287 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:05.283+0000 7fe6c53395c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-01-25T05:42:05.287 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:05.283+0000 7fe6c53395c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-01-25T05:42:05.287 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:05.284+0000 7fe6c53395c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-01-25T05:42:05.810 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:05.808+0000 7fe6c53395c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T05:42:06.570 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:06.567+0000 7fe6c53395c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T05:42:07.357 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-01-25T05:42:07.386 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-25T05:42:07.473 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:07.470+0000 7fead61ae5c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-01-25T05:42:07.473 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:07.470+0000 7fead61ae5c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-01-25T05:42:07.474 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:07.470+0000 7fead61ae5c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-01-25T05:42:07.474 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:07.470+0000 7fead61ae5c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-01-25T05:42:07.474 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:07.470+0000 7fead61ae5c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-01-25T05:42:07.474 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:07.471+0000 7fead61ae5c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-01-25T05:42:08.000 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:07.998+0000 7fead61ae5c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T05:42:08.769 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:08.767+0000 7fead61ae5c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T05:42:09.564 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-01-25T05:42:09.597 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-25T05:42:09.689 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:09.686+0000 7f0a21c575c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-01-25T05:42:09.690 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:09.687+0000 7f0a21c575c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-01-25T05:42:09.690 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:09.687+0000 7f0a21c575c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-01-25T05:42:09.690 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:09.687+0000 7f0a21c575c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-01-25T05:42:09.690 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:09.687+0000 7f0a21c575c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-01-25T05:42:09.690 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:09.687+0000 7f0a21c575c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-01-25T05:42:10.222 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:10.220+0000 7f0a21c575c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T05:42:10.985 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:10.983+0000 7f0a21c575c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T05:42:11.777 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-01-25T05:42:11.809 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-25T05:42:11.904 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:11.901+0000 7fe87aabd5c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-01-25T05:42:11.904 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:11.901+0000 7fe87aabd5c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-01-25T05:42:11.905 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:11.901+0000 7fe87aabd5c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-01-25T05:42:11.905 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:11.901+0000 7fe87aabd5c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-01-25T05:42:11.905 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:11.901+0000 7fe87aabd5c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-01-25T05:42:11.905 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:11.902+0000 7fe87aabd5c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-01-25T05:42:12.428 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:12.426+0000 7fe87aabd5c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T05:42:13.190 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:13.188+0000 7fe87aabd5c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T05:42:13.977 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-01-25T05:42:14.006 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-25T05:42:14.093 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:14.091+0000 7fe080df25c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2023-01-25T05:42:14.094 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:14.091+0000 7fe080df25c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2023-01-25T05:42:14.094 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:14.091+0000 7fe080df25c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2023-01-25T05:42:14.094 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:14.091+0000 7fe080df25c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2023-01-25T05:42:14.094 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:14.091+0000 7fe080df25c0 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-8/block: (2) No such file or directory 2023-01-25T05:42:14.094 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:14.091+0000 7fe080df25c0 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2023-01-25T05:42:14.617 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:14.615+0000 7fe080df25c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T05:42:15.384 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:15.382+0000 7fe080df25c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T05:42:16.176 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2023-01-25T05:42:16.205 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-25T05:42:16.295 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:16.292+0000 7f476516c5c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2023-01-25T05:42:16.295 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:16.293+0000 7f476516c5c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2023-01-25T05:42:16.296 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:16.293+0000 7f476516c5c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2023-01-25T05:42:16.296 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:16.293+0000 7f476516c5c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2023-01-25T05:42:16.296 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:16.293+0000 7f476516c5c0 -1 bluestore(/var/lib/ceph/osd/ceph-9/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-9/block: (2) No such file or directory 2023-01-25T05:42:16.296 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:16.293+0000 7f476516c5c0 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2023-01-25T05:42:16.818 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:16.816+0000 7f476516c5c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T05:42:17.579 INFO:teuthology.orchestra.run.smithi203.stderr:2023-01-25T05:42:17.577+0000 7f476516c5c0 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-01-25T05:42:18.369 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2023-01-25T05:42:18.400 INFO:tasks.ceph:Reading keys from all nodes... 2023-01-25T05:42:18.401 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:42:18.401 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-01-25T05:42:18.470 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:42:18.470 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-01-25T05:42:18.538 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:42:18.538 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-01-25T05:42:18.605 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:42:18.606 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-01-25T05:42:18.677 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:42:18.677 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-01-25T05:42:18.744 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:42:18.744 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-01-25T05:42:18.811 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:42:18.811 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-01-25T05:42:18.878 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:42:18.879 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-01-25T05:42:18.946 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:42:18.946 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-01-25T05:42:19.013 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:42:19.013 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2023-01-25T05:42:19.080 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:42:19.081 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2023-01-25T05:42:19.147 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:42:19.147 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-01-25T05:42:19.205 INFO:tasks.ceph:Adding keys to all mons... 2023-01-25T05:42:19.206 DEBUG:teuthology.orchestra.run.smithi203:> sudo tee -a /etc/ceph/ceph.keyring 2023-01-25T05:42:19.272 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr.x] 2023-01-25T05:42:19.272 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAgwdBjjUD1NxAAbVdH4hP7n4Y9D33n01FpfQ== 2023-01-25T05:42:19.273 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.0] 2023-01-25T05:42:19.273 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAkwdBjyUVvHhAAIjTl+V98RETvzqMKqZKmHw== 2023-01-25T05:42:19.273 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.1] 2023-01-25T05:42:19.273 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAmwdBjGmsVKhAAzyTIzgEO+H6HS4AhuY5LxQ== 2023-01-25T05:42:19.273 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.2] 2023-01-25T05:42:19.274 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAowdBjSZg/NRAAxxFYHB0J3rkN5icSW97ZwQ== 2023-01-25T05:42:19.274 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.3] 2023-01-25T05:42:19.274 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQArwdBjy0qZBRAAq8GDNFHq5//ixEsVF1oCXw== 2023-01-25T05:42:19.274 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.4] 2023-01-25T05:42:19.274 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAtwdBjoLMDERAA0a6iHs0x1qDo4e2Xh7dmhA== 2023-01-25T05:42:19.275 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.5] 2023-01-25T05:42:19.275 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAvwdBjE88pHBAACZJs9TSn0W38hmuiPiSqaw== 2023-01-25T05:42:19.275 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.6] 2023-01-25T05:42:19.275 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAxwdBjQgIMKRAAQihohY2s/KBxrx/baWN04g== 2023-01-25T05:42:19.275 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.7] 2023-01-25T05:42:19.276 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAzwdBjn7LZNRAAmq2tgLUeJLEYvhdGGy774g== 2023-01-25T05:42:19.276 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.8] 2023-01-25T05:42:19.276 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQA2wdBj6GWHBRAAIurHfnTXcM032rxUbpWdaQ== 2023-01-25T05:42:19.276 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.9] 2023-01-25T05:42:19.276 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQA4wdBjd2uPERAAPkUNdCqnGW/le8I4WBkWcA== 2023-01-25T05:42:19.277 INFO:teuthology.orchestra.run.smithi203.stdout:[client.0] 2023-01-25T05:42:19.277 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAgwdBjsZcCOxAA3ZhMZs6LJogOhCSc4J+W2g== 2023-01-25T05:42:19.278 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-01-25T05:42:19.371 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T05:42:19.425 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T05:42:19.479 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T05:42:19.532 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T05:42:19.586 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T05:42:19.640 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T05:42:19.693 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T05:42:19.747 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T05:42:19.801 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T05:42:19.855 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-25T05:42:19.909 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-01-25T05:42:19.965 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-01-25T05:42:19.966 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-01-25T05:42:19.993 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-01-25T05:42:20.091 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-01-25T05:42:20.118 DEBUG:teuthology.orchestra.run.smithi203:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-01-25T05:42:20.176 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-01-25T05:42:20.177 INFO:tasks.ceph.mon.a:Restarting daemon 2023-01-25T05:42:20.177 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-01-25T05:42:20.219 INFO:tasks.ceph.mon.a:Started 2023-01-25T05:42:20.219 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph config set mgr mgr_pool false --force 2023-01-25T05:42:20.447 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-01-25T05:42:20.448 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-01-25T05:42:20.448 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-01-25T05:42:20.450 INFO:tasks.ceph.mgr.x:Started 2023-01-25T05:42:20.450 INFO:tasks.ceph:Setting crush tunables to default 2023-01-25T05:42:20.451 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd crush tunables default 2023-01-25T05:42:20.660 INFO:teuthology.orchestra.run.smithi203.stderr:adjusted tunables profile to default 2023-01-25T05:42:20.671 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-01-25T05:42:20.672 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:42:20.672 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-01-25T05:42:20.699 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:42:20.700 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-01-25T05:42:20.770 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:42:20.771 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-01-25T05:42:20.837 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:42:20.838 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-01-25T05:42:20.905 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:42:20.905 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-01-25T05:42:20.972 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:42:20.973 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-01-25T05:42:21.040 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:42:21.041 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-01-25T05:42:21.107 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:42:21.107 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-01-25T05:42:21.174 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:42:21.175 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2023-01-25T05:42:21.242 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:42:21.243 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2023-01-25T05:42:21.311 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new f87dc56d-86e7-496d-8e8f-b0d7dbca30a4 0 2023-01-25T05:42:21.454 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:21.453+0000 7f8eaabe7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-01-25T05:42:21.527 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:21.526+0000 7f8eaabe7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-01-25T05:42:21.566 INFO:teuthology.orchestra.run.smithi203.stdout:0 2023-01-25T05:42:21.579 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new f35f7666-378a-4cc5-bd86-fd0f0a2e1b31 1 2023-01-25T05:42:21.611 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:21.610+0000 7f8eaabe7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-01-25T05:42:21.793 INFO:teuthology.orchestra.run.smithi203.stdout:1 2023-01-25T05:42:21.804 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:21.803+0000 7f8eaabe7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-01-25T05:42:21.805 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new e5ea1037-85eb-494e-b195-161cd77cb6fb 2 2023-01-25T05:42:22.060 INFO:teuthology.orchestra.run.smithi203.stdout:2 2023-01-25T05:42:22.073 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 3451afca-3f67-4a33-88bc-9c4a3fa71b37 3 2023-01-25T05:42:22.286 INFO:teuthology.orchestra.run.smithi203.stdout:3 2023-01-25T05:42:22.300 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 53249305-7472-4032-88c6-b6f2eab2f939 4 2023-01-25T05:42:22.561 INFO:teuthology.orchestra.run.smithi203.stdout:4 2023-01-25T05:42:22.575 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 3c6cd644-1d97-4a74-815a-cbffa9685718 5 2023-01-25T05:42:22.652 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:22.650+0000 7f8eaabe7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-01-25T05:42:22.791 INFO:teuthology.orchestra.run.smithi203.stdout:5 2023-01-25T05:42:22.799 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:22.798+0000 7f8eaabe7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-01-25T05:42:22.804 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 1426af76-6197-499f-8aa3-9202c91b6fc7 6 2023-01-25T05:42:23.022 INFO:teuthology.orchestra.run.smithi203.stdout:6 2023-01-25T05:42:23.035 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new a13fb8e1-a46d-4398-956d-ce4d7b04da48 7 2023-01-25T05:42:23.038 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:23.036+0000 7f8eaabe7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-01-25T05:42:23.113 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:23.111+0000 7f8eaabe7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-01-25T05:42:23.250 INFO:teuthology.orchestra.run.smithi203.stdout:7 2023-01-25T05:42:23.253 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:23.252+0000 7f8eaabe7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-01-25T05:42:23.262 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new cc46f5f4-d476-4ff6-b7f9-537263cb24cd 8 2023-01-25T05:42:23.475 INFO:teuthology.orchestra.run.smithi203.stdout:8 2023-01-25T05:42:23.488 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 2ac16a61-4dcd-43be-a686-f9959a8901ea 9 2023-01-25T05:42:23.532 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:23.531+0000 7f8eaabe7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-01-25T05:42:23.707 INFO:teuthology.orchestra.run.smithi203.stdout:9 2023-01-25T05:42:23.720 INFO:tasks.ceph.osd.0:Restarting daemon 2023-01-25T05:42:23.720 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-01-25T05:42:23.722 INFO:tasks.ceph.osd.0:Started 2023-01-25T05:42:23.723 INFO:tasks.ceph.osd.1:Restarting daemon 2023-01-25T05:42:23.723 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-01-25T05:42:23.725 INFO:tasks.ceph.osd.1:Started 2023-01-25T05:42:23.726 INFO:tasks.ceph.osd.2:Restarting daemon 2023-01-25T05:42:23.726 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-01-25T05:42:23.727 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:23.727+0000 7f8eaabe7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-01-25T05:42:23.728 INFO:tasks.ceph.osd.2:Started 2023-01-25T05:42:23.729 INFO:tasks.ceph.osd.3:Restarting daemon 2023-01-25T05:42:23.729 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-01-25T05:42:23.731 INFO:tasks.ceph.osd.3:Started 2023-01-25T05:42:23.732 INFO:tasks.ceph.osd.4:Restarting daemon 2023-01-25T05:42:23.732 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-01-25T05:42:23.734 INFO:tasks.ceph.osd.4:Started 2023-01-25T05:42:23.735 INFO:tasks.ceph.osd.5:Restarting daemon 2023-01-25T05:42:23.735 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-01-25T05:42:23.739 INFO:tasks.ceph.osd.5:Started 2023-01-25T05:42:23.740 INFO:tasks.ceph.osd.6:Restarting daemon 2023-01-25T05:42:23.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-01-25T05:42:23.745 INFO:tasks.ceph.osd.6:Started 2023-01-25T05:42:23.745 INFO:tasks.ceph.osd.7:Restarting daemon 2023-01-25T05:42:23.746 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-01-25T05:42:23.772 INFO:tasks.ceph.osd.7:Started 2023-01-25T05:42:23.772 INFO:tasks.ceph.osd.8:Restarting daemon 2023-01-25T05:42:23.773 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2023-01-25T05:42:23.780 INFO:tasks.ceph.osd.8:Started 2023-01-25T05:42:23.781 INFO:tasks.ceph.osd.9:Restarting daemon 2023-01-25T05:42:23.781 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2023-01-25T05:42:23.787 INFO:tasks.ceph.osd.9:Started 2023-01-25T05:42:23.788 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-25T05:42:23.958 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:23.956+0000 7f8eaabe7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-01-25T05:42:24.078 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:24.077+0000 7f8eaabe7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-01-25T05:42:24.093 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:42:24.093 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":12,"fsid":"5b00363d-2783-4934-a70f-2b1720d57fe5","created":"2023-01-25T05:42:20.347257+0000","modified":"2023-01-25T05:42:23.705123+0000","last_up_change":"0.000000","last_in_change":"2023-01-25T05:42:23.705123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[],"osds":[{"osd":0,"uuid":"f87dc56d-86e7-496d-8e8f-b0d7dbca30a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f35f7666-378a-4cc5-bd86-fd0f0a2e1b31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5ea1037-85eb-494e-b195-161cd77cb6fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3451afca-3f67-4a33-88bc-9c4a3fa71b37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53249305-7472-4032-88c6-b6f2eab2f939","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c6cd644-1d97-4a74-815a-cbffa9685718","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1426af76-6197-499f-8aa3-9202c91b6fc7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a13fb8e1-a46d-4398-956d-ce4d7b04da48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"cc46f5f4-d476-4ff6-b7f9-537263cb24cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"2ac16a61-4dcd-43be-a686-f9959a8901ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-25T05:42:24.107 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-01-25T05:42:24.107 INFO:tasks.ceph:Waiting for OSDs to come up 2023-01-25T05:42:24.172 INFO:tasks.ceph.osd.3.smithi203.stderr:2023-01-25T05:42:24.171+0000 7f0577d6a5c0 -1 Falling back to public interface 2023-01-25T05:42:24.173 INFO:tasks.ceph.osd.0.smithi203.stderr:2023-01-25T05:42:24.172+0000 7f23d92b25c0 -1 Falling back to public interface 2023-01-25T05:42:24.221 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:24.220+0000 7f8eaabe7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-01-25T05:42:24.301 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:24.300+0000 7f8eaabe7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-01-25T05:42:24.395 INFO:tasks.ceph.osd.2.smithi203.stderr:2023-01-25T05:42:24.394+0000 7f10e5f4a5c0 -1 Falling back to public interface 2023-01-25T05:42:24.399 INFO:tasks.ceph.osd.1.smithi203.stderr:2023-01-25T05:42:24.398+0000 7f12a98ac5c0 -1 Falling back to public interface 2023-01-25T05:42:24.417 INFO:tasks.ceph.osd.5.smithi203.stderr:2023-01-25T05:42:24.416+0000 7f6cc7eb75c0 -1 Falling back to public interface 2023-01-25T05:42:24.420 INFO:tasks.ceph.osd.6.smithi203.stderr:2023-01-25T05:42:24.419+0000 7f0d4a1fb5c0 -1 Falling back to public interface 2023-01-25T05:42:24.423 INFO:tasks.ceph.osd.4.smithi203.stderr:2023-01-25T05:42:24.422+0000 7f98696b35c0 -1 Falling back to public interface 2023-01-25T05:42:24.467 INFO:tasks.ceph.osd.8.smithi203.stderr:2023-01-25T05:42:24.466+0000 7f75ce6915c0 -1 Falling back to public interface 2023-01-25T05:42:24.471 INFO:tasks.ceph.osd.7.smithi203.stderr:2023-01-25T05:42:24.470+0000 7f3451eb65c0 -1 Falling back to public interface 2023-01-25T05:42:24.474 INFO:tasks.ceph.osd.7.smithi203.stderr:2023-01-25T05:42:24.473+0000 7f3449264700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6863/0: (98) Address already in use 2023-01-25T05:42:24.483 INFO:tasks.ceph.osd.9.smithi203.stderr:2023-01-25T05:42:24.482+0000 7fe3296a95c0 -1 Falling back to public interface 2023-01-25T05:42:24.531 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:24.530+0000 7f8eaabe7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-01-25T05:42:24.609 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:24.607+0000 7f8eaabe7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-01-25T05:42:25.113 DEBUG:teuthology.orchestra.run.smithi203:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-01-25T05:42:25.239 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:25.238+0000 7f8eaabe7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-01-25T05:42:25.307 INFO:teuthology.misc.health.smithi203.stdout: 2023-01-25T05:42:25.308 INFO:teuthology.misc.health.smithi203.stdout:{"epoch":12,"fsid":"5b00363d-2783-4934-a70f-2b1720d57fe5","created":"2023-01-25T05:42:20.347257+0000","modified":"2023-01-25T05:42:23.705123+0000","last_up_change":"0.000000","last_in_change":"2023-01-25T05:42:23.705123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[],"osds":[{"osd":0,"uuid":"f87dc56d-86e7-496d-8e8f-b0d7dbca30a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f35f7666-378a-4cc5-bd86-fd0f0a2e1b31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e5ea1037-85eb-494e-b195-161cd77cb6fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3451afca-3f67-4a33-88bc-9c4a3fa71b37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"53249305-7472-4032-88c6-b6f2eab2f939","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3c6cd644-1d97-4a74-815a-cbffa9685718","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1426af76-6197-499f-8aa3-9202c91b6fc7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a13fb8e1-a46d-4398-956d-ce4d7b04da48","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"cc46f5f4-d476-4ff6-b7f9-537263cb24cd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"2ac16a61-4dcd-43be-a686-f9959a8901ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-25T05:42:25.319 DEBUG:teuthology.misc:0 of 10 OSDs are up 2023-01-25T05:42:25.499 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:25.498+0000 7f8eaabe7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-01-25T05:42:25.577 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:25.576+0000 7f8eaabe7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-01-25T05:42:25.659 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:25.658+0000 7f8eaabe7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-01-25T05:42:25.847 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:25.846+0000 7f8eaabe7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-01-25T05:42:25.919 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:25.918+0000 7f8eaabe7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-01-25T05:42:26.073 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:26.072+0000 7f8eaabe7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-01-25T05:42:26.923 INFO:tasks.ceph.osd.0.smithi203.stderr:2023-01-25T05:42:26.922+0000 7f23d92b25c0 -1 osd.0 0 log_to_monitors true 2023-01-25T05:42:26.924 INFO:tasks.ceph.osd.3.smithi203.stderr:2023-01-25T05:42:26.922+0000 7f0577d6a5c0 -1 osd.3 0 log_to_monitors true 2023-01-25T05:42:27.091 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-01-25T05:42:27.090+0000 7f8e64727700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-01-25T05:42:27.126 INFO:tasks.ceph.osd.2.smithi203.stderr:2023-01-25T05:42:27.125+0000 7f10e5f4a5c0 -1 osd.2 0 log_to_monitors true 2023-01-25T05:42:27.131 INFO:tasks.ceph.osd.1.smithi203.stderr:2023-01-25T05:42:27.130+0000 7f12a98ac5c0 -1 osd.1 0 log_to_monitors true 2023-01-25T05:42:27.162 INFO:tasks.ceph.osd.5.smithi203.stderr:2023-01-25T05:42:27.161+0000 7f6cc7eb75c0 -1 osd.5 0 log_to_monitors true 2023-01-25T05:42:27.184 INFO:tasks.ceph.osd.6.smithi203.stderr:2023-01-25T05:42:27.183+0000 7f0d4a1fb5c0 -1 osd.6 0 log_to_monitors true 2023-01-25T05:42:27.192 INFO:tasks.ceph.osd.4.smithi203.stderr:2023-01-25T05:42:27.191+0000 7f98696b35c0 -1 osd.4 0 log_to_monitors true 2023-01-25T05:42:27.207 INFO:tasks.ceph.osd.7.smithi203.stderr:2023-01-25T05:42:27.206+0000 7f3451eb65c0 -1 osd.7 0 log_to_monitors true 2023-01-25T05:42:27.209 INFO:tasks.ceph.osd.8.smithi203.stderr:2023-01-25T05:42:27.206+0000 7f75ce6915c0 -1 osd.8 0 log_to_monitors true 2023-01-25T05:42:27.227 INFO:tasks.ceph.osd.9.smithi203.stderr:2023-01-25T05:42:27.226+0000 7fe3296a95c0 -1 osd.9 0 log_to_monitors true 2023-01-25T05:42:28.334 INFO:tasks.ceph.osd.1.smithi203.stderr:2023-01-25T05:42:28.333+0000 7f129dc54700 -1 osd.1 0 waiting for initial osdmap 2023-01-25T05:42:28.334 INFO:tasks.ceph.osd.4.smithi203.stderr:2023-01-25T05:42:28.333+0000 7f985da5b700 -1 osd.4 0 waiting for initial osdmap 2023-01-25T05:42:28.335 INFO:tasks.ceph.osd.6.smithi203.stderr:2023-01-25T05:42:28.334+0000 7f0d3e5a3700 -1 osd.6 0 waiting for initial osdmap 2023-01-25T05:42:28.335 INFO:tasks.ceph.osd.7.smithi203.stderr:2023-01-25T05:42:28.334+0000 7f344625e700 -1 osd.7 0 waiting for initial osdmap 2023-01-25T05:42:28.335 INFO:tasks.ceph.osd.2.smithi203.stderr:2023-01-25T05:42:28.334+0000 7f10da2f2700 -1 osd.2 0 waiting for initial osdmap 2023-01-25T05:42:28.336 INFO:tasks.ceph.osd.0.smithi203.stderr:2023-01-25T05:42:28.334+0000 7f23cd65a700 -1 osd.0 0 waiting for initial osdmap 2023-01-25T05:42:28.336 INFO:tasks.ceph.osd.8.smithi203.stderr:2023-01-25T05:42:28.335+0000 7f75c2a39700 -1 osd.8 0 waiting for initial osdmap 2023-01-25T05:42:28.337 INFO:tasks.ceph.osd.4.smithi203.stderr:2023-01-25T05:42:28.337+0000 7f9857639700 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:42:28.338 INFO:tasks.ceph.osd.1.smithi203.stderr:2023-01-25T05:42:28.337+0000 7f1297832700 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:42:28.339 INFO:tasks.ceph.osd.9.smithi203.stderr:2023-01-25T05:42:28.338+0000 7fe31da51700 -1 osd.9 0 waiting for initial osdmap 2023-01-25T05:42:28.339 INFO:tasks.ceph.osd.5.smithi203.stderr:2023-01-25T05:42:28.338+0000 7f6cbc25f700 -1 osd.5 0 waiting for initial osdmap 2023-01-25T05:42:28.339 INFO:tasks.ceph.osd.6.smithi203.stderr:2023-01-25T05:42:28.339+0000 7f0d38181700 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:42:28.340 INFO:tasks.ceph.osd.0.smithi203.stderr:2023-01-25T05:42:28.339+0000 7f23c7a39700 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:42:28.340 INFO:tasks.ceph.osd.2.smithi203.stderr:2023-01-25T05:42:28.339+0000 7f10d3ed0700 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:42:28.341 INFO:tasks.ceph.osd.8.smithi203.stderr:2023-01-25T05:42:28.340+0000 7f75bc617700 -1 osd.8 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:42:28.341 INFO:tasks.ceph.osd.7.smithi203.stderr:2023-01-25T05:42:28.340+0000 7f343fe3c700 -1 osd.7 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:42:28.342 INFO:tasks.ceph.osd.3.smithi203.stderr:2023-01-25T05:42:28.341+0000 7f056c112700 -1 osd.3 0 waiting for initial osdmap 2023-01-25T05:42:28.342 INFO:tasks.ceph.osd.5.smithi203.stderr:2023-01-25T05:42:28.341+0000 7f6cb5e3d700 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:42:28.342 INFO:tasks.ceph.osd.9.smithi203.stderr:2023-01-25T05:42:28.342+0000 7fe31762f700 -1 osd.9 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:42:28.345 INFO:tasks.ceph.osd.3.smithi203.stderr:2023-01-25T05:42:28.344+0000 7f05664f1700 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:42:32.325 DEBUG:teuthology.orchestra.run.smithi203:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-01-25T05:42:32.641 INFO:teuthology.misc.health.smithi203.stdout: 2023-01-25T05:42:32.642 INFO:teuthology.misc.health.smithi203.stdout:{"epoch":16,"fsid":"5b00363d-2783-4934-a70f-2b1720d57fe5","created":"2023-01-25T05:42:20.347257+0000","modified":"2023-01-25T05:42:30.339171+0000","last_up_change":"2023-01-25T05:42:29.316228+0000","last_in_change":"2023-01-25T05:42:23.705123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[],"osds":[{"osd":0,"uuid":"f87dc56d-86e7-496d-8e8f-b0d7dbca30a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":110629},{"type":"v1","addr":"172.21.15.203:6809","nonce":110629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":110629},{"type":"v1","addr":"172.21.15.203:6811","nonce":110629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":110629},{"type":"v1","addr":"172.21.15.203:6815","nonce":110629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":110629},{"type":"v1","addr":"172.21.15.203:6813","nonce":110629}]},"public_addr":"172.21.15.203:6809/110629","cluster_addr":"172.21.15.203:6811/110629","heartbeat_back_addr":"172.21.15.203:6815/110629","heartbeat_front_addr":"172.21.15.203:6813/110629","state":["exists","up"]},{"osd":1,"uuid":"f35f7666-378a-4cc5-bd86-fd0f0a2e1b31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":110637},{"type":"v1","addr":"172.21.15.203:6825","nonce":110637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":110637},{"type":"v1","addr":"172.21.15.203:6827","nonce":110637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":110637},{"type":"v1","addr":"172.21.15.203:6831","nonce":110637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":110637},{"type":"v1","addr":"172.21.15.203:6829","nonce":110637}]},"public_addr":"172.21.15.203:6825/110637","cluster_addr":"172.21.15.203:6827/110637","heartbeat_back_addr":"172.21.15.203:6831/110637","heartbeat_front_addr":"172.21.15.203:6829/110637","state":["exists","up"]},{"osd":2,"uuid":"e5ea1037-85eb-494e-b195-161cd77cb6fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":110630},{"type":"v1","addr":"172.21.15.203:6817","nonce":110630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":110630},{"type":"v1","addr":"172.21.15.203:6819","nonce":110630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":110630},{"type":"v1","addr":"172.21.15.203:6823","nonce":110630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":110630},{"type":"v1","addr":"172.21.15.203:6821","nonce":110630}]},"public_addr":"172.21.15.203:6817/110630","cluster_addr":"172.21.15.203:6819/110630","heartbeat_back_addr":"172.21.15.203:6823/110630","heartbeat_front_addr":"172.21.15.203:6821/110630","state":["exists","up"]},{"osd":3,"uuid":"3451afca-3f67-4a33-88bc-9c4a3fa71b37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":110642},{"type":"v1","addr":"172.21.15.203:6801","nonce":110642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":110642},{"type":"v1","addr":"172.21.15.203:6803","nonce":110642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":110642},{"type":"v1","addr":"172.21.15.203:6807","nonce":110642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":110642},{"type":"v1","addr":"172.21.15.203:6805","nonce":110642}]},"public_addr":"172.21.15.203:6801/110642","cluster_addr":"172.21.15.203:6803/110642","heartbeat_back_addr":"172.21.15.203:6807/110642","heartbeat_front_addr":"172.21.15.203:6805/110642","state":["exists","up"]},{"osd":4,"uuid":"53249305-7472-4032-88c6-b6f2eab2f939","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6845","nonce":110696},{"type":"v1","addr":"172.21.15.203:6847","nonce":110696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6849","nonce":110696},{"type":"v1","addr":"172.21.15.203:6851","nonce":110696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6854","nonce":110696},{"type":"v1","addr":"172.21.15.203:6855","nonce":110696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6852","nonce":110696},{"type":"v1","addr":"172.21.15.203:6853","nonce":110696}]},"public_addr":"172.21.15.203:6847/110696","cluster_addr":"172.21.15.203:6851/110696","heartbeat_back_addr":"172.21.15.203:6855/110696","heartbeat_front_addr":"172.21.15.203:6853/110696","state":["exists","up"]},{"osd":5,"uuid":"3c6cd644-1d97-4a74-815a-cbffa9685718","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":110644},{"type":"v1","addr":"172.21.15.203:6833","nonce":110644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":110644},{"type":"v1","addr":"172.21.15.203:6835","nonce":110644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":110644},{"type":"v1","addr":"172.21.15.203:6840","nonce":110644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":110644},{"type":"v1","addr":"172.21.15.203:6837","nonce":110644}]},"public_addr":"172.21.15.203:6833/110644","cluster_addr":"172.21.15.203:6835/110644","heartbeat_back_addr":"172.21.15.203:6840/110644","heartbeat_front_addr":"172.21.15.203:6837/110644","state":["exists","up"]},{"osd":6,"uuid":"1426af76-6197-499f-8aa3-9202c91b6fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6839","nonce":110685},{"type":"v1","addr":"172.21.15.203:6841","nonce":110685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6842","nonce":110685},{"type":"v1","addr":"172.21.15.203:6843","nonce":110685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6848","nonce":110685},{"type":"v1","addr":"172.21.15.203:6850","nonce":110685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6844","nonce":110685},{"type":"v1","addr":"172.21.15.203:6846","nonce":110685}]},"public_addr":"172.21.15.203:6841/110685","cluster_addr":"172.21.15.203:6843/110685","heartbeat_back_addr":"172.21.15.203:6850/110685","heartbeat_front_addr":"172.21.15.203:6846/110685","state":["exists","up"]},{"osd":7,"uuid":"a13fb8e1-a46d-4398-956d-ce4d7b04da48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6859","nonce":110783},{"type":"v1","addr":"172.21.15.203:6861","nonce":110783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6864","nonce":110783},{"type":"v1","addr":"172.21.15.203:6866","nonce":110783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6870","nonce":110783},{"type":"v1","addr":"172.21.15.203:6871","nonce":110783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6868","nonce":110783},{"type":"v1","addr":"172.21.15.203:6869","nonce":110783}]},"public_addr":"172.21.15.203:6861/110783","cluster_addr":"172.21.15.203:6866/110783","heartbeat_back_addr":"172.21.15.203:6871/110783","heartbeat_front_addr":"172.21.15.203:6869/110783","state":["exists","up"]},{"osd":8,"uuid":"cc46f5f4-d476-4ff6-b7f9-537263cb24cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6856","nonce":110781},{"type":"v1","addr":"172.21.15.203:6857","nonce":110781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6858","nonce":110781},{"type":"v1","addr":"172.21.15.203:6860","nonce":110781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6865","nonce":110781},{"type":"v1","addr":"172.21.15.203:6867","nonce":110781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6862","nonce":110781},{"type":"v1","addr":"172.21.15.203:6863","nonce":110781}]},"public_addr":"172.21.15.203:6857/110781","cluster_addr":"172.21.15.203:6860/110781","heartbeat_back_addr":"172.21.15.203:6867/110781","heartbeat_front_addr":"172.21.15.203:6863/110781","state":["exists","up"]},{"osd":9,"uuid":"2ac16a61-4dcd-43be-a686-f9959a8901ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6872","nonce":110792},{"type":"v1","addr":"172.21.15.203:6873","nonce":110792}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6874","nonce":110792},{"type":"v1","addr":"172.21.15.203:6875","nonce":110792}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6878","nonce":110792},{"type":"v1","addr":"172.21.15.203:6879","nonce":110792}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6876","nonce":110792},{"type":"v1","addr":"172.21.15.203:6877","nonce":110792}]},"public_addr":"172.21.15.203:6873/110792","cluster_addr":"172.21.15.203:6875/110792","heartbeat_back_addr":"172.21.15.203:6879/110792","heartbeat_front_addr":"172.21.15.203:6877/110792","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:27.924170+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.162198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.104251+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:27.901694+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.209516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.128175+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.144209+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.228683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.195999+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.258473+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-25T05:42:32.655 DEBUG:teuthology.misc:10 of 10 OSDs are up 2023-01-25T05:42:32.655 INFO:tasks.ceph:Creating RBD pool 2023-01-25T05:42:32.656 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd pool create rbd 8 2023-01-25T05:42:33.350 INFO:teuthology.orchestra.run.smithi203.stderr:pool 'rbd' created 2023-01-25T05:42:33.362 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-01-25T05:42:34.359 INFO:teuthology.orchestra.run.smithi203.stderr:enabled application 'rbd' on pool 'rbd' 2023-01-25T05:42:34.371 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-01-25T05:42:34.372 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-01-25T05:42:34.372 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-01-25T05:42:34.373 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-01-25T05:42:34.373 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-01-25T05:42:34.744 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:42:34.757 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6880","nonce":109620},{"type":"v1","addr":"172.21.15.203:6881","nonce":109620}]},"active_addr":"172.21.15.203:6881/109620","active_change":"2023-01-25T05:42:26.085457+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":3582124343}]},{"addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":2277995203}]},{"addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":1769065259}]}]}} 2023-01-25T05:42:34.759 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-01-25T05:42:34.759 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-01-25T05:42:34.760 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-25T05:42:35.076 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:42:35.076 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":18,"fsid":"5b00363d-2783-4934-a70f-2b1720d57fe5","created":"2023-01-25T05:42:20.347257+0000","modified":"2023-01-25T05:42:34.350445+0000","last_up_change":"2023-01-25T05:42:29.316228+0000","last_in_change":"2023-01-25T05:42:23.705123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-01-25T05:42:32.990588+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f87dc56d-86e7-496d-8e8f-b0d7dbca30a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":110629},{"type":"v1","addr":"172.21.15.203:6809","nonce":110629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":110629},{"type":"v1","addr":"172.21.15.203:6811","nonce":110629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":110629},{"type":"v1","addr":"172.21.15.203:6815","nonce":110629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":110629},{"type":"v1","addr":"172.21.15.203:6813","nonce":110629}]},"public_addr":"172.21.15.203:6809/110629","cluster_addr":"172.21.15.203:6811/110629","heartbeat_back_addr":"172.21.15.203:6815/110629","heartbeat_front_addr":"172.21.15.203:6813/110629","state":["exists","up"]},{"osd":1,"uuid":"f35f7666-378a-4cc5-bd86-fd0f0a2e1b31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":110637},{"type":"v1","addr":"172.21.15.203:6825","nonce":110637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":110637},{"type":"v1","addr":"172.21.15.203:6827","nonce":110637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":110637},{"type":"v1","addr":"172.21.15.203:6831","nonce":110637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":110637},{"type":"v1","addr":"172.21.15.203:6829","nonce":110637}]},"public_addr":"172.21.15.203:6825/110637","cluster_addr":"172.21.15.203:6827/110637","heartbeat_back_addr":"172.21.15.203:6831/110637","heartbeat_front_addr":"172.21.15.203:6829/110637","state":["exists","up"]},{"osd":2,"uuid":"e5ea1037-85eb-494e-b195-161cd77cb6fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":110630},{"type":"v1","addr":"172.21.15.203:6817","nonce":110630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":110630},{"type":"v1","addr":"172.21.15.203:6819","nonce":110630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":110630},{"type":"v1","addr":"172.21.15.203:6823","nonce":110630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":110630},{"type":"v1","addr":"172.21.15.203:6821","nonce":110630}]},"public_addr":"172.21.15.203:6817/110630","cluster_addr":"172.21.15.203:6819/110630","heartbeat_back_addr":"172.21.15.203:6823/110630","heartbeat_front_addr":"172.21.15.203:6821/110630","state":["exists","up"]},{"osd":3,"uuid":"3451afca-3f67-4a33-88bc-9c4a3fa71b37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":110642},{"type":"v1","addr":"172.21.15.203:6801","nonce":110642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":110642},{"type":"v1","addr":"172.21.15.203:6803","nonce":110642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":110642},{"type":"v1","addr":"172.21.15.203:6807","nonce":110642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":110642},{"type":"v1","addr":"172.21.15.203:6805","nonce":110642}]},"public_addr":"172.21.15.203:6801/110642","cluster_addr":"172.21.15.203:6803/110642","heartbeat_back_addr":"172.21.15.203:6807/110642","heartbeat_front_addr":"172.21.15.203:6805/110642","state":["exists","up"]},{"osd":4,"uuid":"53249305-7472-4032-88c6-b6f2eab2f939","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6845","nonce":110696},{"type":"v1","addr":"172.21.15.203:6847","nonce":110696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6849","nonce":110696},{"type":"v1","addr":"172.21.15.203:6851","nonce":110696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6854","nonce":110696},{"type":"v1","addr":"172.21.15.203:6855","nonce":110696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6852","nonce":110696},{"type":"v1","addr":"172.21.15.203:6853","nonce":110696}]},"public_addr":"172.21.15.203:6847/110696","cluster_addr":"172.21.15.203:6851/110696","heartbeat_back_addr":"172.21.15.203:6855/110696","heartbeat_front_addr":"172.21.15.203:6853/110696","state":["exists","up"]},{"osd":5,"uuid":"3c6cd644-1d97-4a74-815a-cbffa9685718","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":110644},{"type":"v1","addr":"172.21.15.203:6833","nonce":110644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":110644},{"type":"v1","addr":"172.21.15.203:6835","nonce":110644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":110644},{"type":"v1","addr":"172.21.15.203:6840","nonce":110644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":110644},{"type":"v1","addr":"172.21.15.203:6837","nonce":110644}]},"public_addr":"172.21.15.203:6833/110644","cluster_addr":"172.21.15.203:6835/110644","heartbeat_back_addr":"172.21.15.203:6840/110644","heartbeat_front_addr":"172.21.15.203:6837/110644","state":["exists","up"]},{"osd":6,"uuid":"1426af76-6197-499f-8aa3-9202c91b6fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6839","nonce":110685},{"type":"v1","addr":"172.21.15.203:6841","nonce":110685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6842","nonce":110685},{"type":"v1","addr":"172.21.15.203:6843","nonce":110685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6848","nonce":110685},{"type":"v1","addr":"172.21.15.203:6850","nonce":110685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6844","nonce":110685},{"type":"v1","addr":"172.21.15.203:6846","nonce":110685}]},"public_addr":"172.21.15.203:6841/110685","cluster_addr":"172.21.15.203:6843/110685","heartbeat_back_addr":"172.21.15.203:6850/110685","heartbeat_front_addr":"172.21.15.203:6846/110685","state":["exists","up"]},{"osd":7,"uuid":"a13fb8e1-a46d-4398-956d-ce4d7b04da48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6859","nonce":110783},{"type":"v1","addr":"172.21.15.203:6861","nonce":110783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6864","nonce":110783},{"type":"v1","addr":"172.21.15.203:6866","nonce":110783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6870","nonce":110783},{"type":"v1","addr":"172.21.15.203:6871","nonce":110783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6868","nonce":110783},{"type":"v1","addr":"172.21.15.203:6869","nonce":110783}]},"public_addr":"172.21.15.203:6861/110783","cluster_addr":"172.21.15.203:6866/110783","heartbeat_back_addr":"172.21.15.203:6871/110783","heartbeat_front_addr":"172.21.15.203:6869/110783","state":["exists","up"]},{"osd":8,"uuid":"cc46f5f4-d476-4ff6-b7f9-537263cb24cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6856","nonce":110781},{"type":"v1","addr":"172.21.15.203:6857","nonce":110781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6858","nonce":110781},{"type":"v1","addr":"172.21.15.203:6860","nonce":110781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6865","nonce":110781},{"type":"v1","addr":"172.21.15.203:6867","nonce":110781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6862","nonce":110781},{"type":"v1","addr":"172.21.15.203:6863","nonce":110781}]},"public_addr":"172.21.15.203:6857/110781","cluster_addr":"172.21.15.203:6860/110781","heartbeat_back_addr":"172.21.15.203:6867/110781","heartbeat_front_addr":"172.21.15.203:6863/110781","state":["exists","up"]},{"osd":9,"uuid":"2ac16a61-4dcd-43be-a686-f9959a8901ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6872","nonce":110792},{"type":"v1","addr":"172.21.15.203:6873","nonce":110792}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6874","nonce":110792},{"type":"v1","addr":"172.21.15.203:6875","nonce":110792}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6878","nonce":110792},{"type":"v1","addr":"172.21.15.203:6879","nonce":110792}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6876","nonce":110792},{"type":"v1","addr":"172.21.15.203:6877","nonce":110792}]},"public_addr":"172.21.15.203:6873/110792","cluster_addr":"172.21.15.203:6875/110792","heartbeat_back_addr":"172.21.15.203:6879/110792","heartbeat_front_addr":"172.21.15.203:6877/110792","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:27.924170+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.162198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.104251+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:27.901694+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.209516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.128175+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.144209+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.228683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.195999+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.258473+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-25T05:42:35.090 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-01-25T05:42:35.090 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-25T05:42:35.403 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:42:35.404 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":19,"fsid":"5b00363d-2783-4934-a70f-2b1720d57fe5","created":"2023-01-25T05:42:20.347257+0000","modified":"2023-01-25T05:42:35.352774+0000","last_up_change":"2023-01-25T05:42:29.316228+0000","last_in_change":"2023-01-25T05:42:23.705123+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-01-25T05:42:32.990588+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f87dc56d-86e7-496d-8e8f-b0d7dbca30a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":110629},{"type":"v1","addr":"172.21.15.203:6809","nonce":110629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":110629},{"type":"v1","addr":"172.21.15.203:6811","nonce":110629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":110629},{"type":"v1","addr":"172.21.15.203:6815","nonce":110629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":110629},{"type":"v1","addr":"172.21.15.203:6813","nonce":110629}]},"public_addr":"172.21.15.203:6809/110629","cluster_addr":"172.21.15.203:6811/110629","heartbeat_back_addr":"172.21.15.203:6815/110629","heartbeat_front_addr":"172.21.15.203:6813/110629","state":["exists","up"]},{"osd":1,"uuid":"f35f7666-378a-4cc5-bd86-fd0f0a2e1b31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":110637},{"type":"v1","addr":"172.21.15.203:6825","nonce":110637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":110637},{"type":"v1","addr":"172.21.15.203:6827","nonce":110637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":110637},{"type":"v1","addr":"172.21.15.203:6831","nonce":110637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":110637},{"type":"v1","addr":"172.21.15.203:6829","nonce":110637}]},"public_addr":"172.21.15.203:6825/110637","cluster_addr":"172.21.15.203:6827/110637","heartbeat_back_addr":"172.21.15.203:6831/110637","heartbeat_front_addr":"172.21.15.203:6829/110637","state":["exists","up"]},{"osd":2,"uuid":"e5ea1037-85eb-494e-b195-161cd77cb6fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":110630},{"type":"v1","addr":"172.21.15.203:6817","nonce":110630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":110630},{"type":"v1","addr":"172.21.15.203:6819","nonce":110630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":110630},{"type":"v1","addr":"172.21.15.203:6823","nonce":110630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":110630},{"type":"v1","addr":"172.21.15.203:6821","nonce":110630}]},"public_addr":"172.21.15.203:6817/110630","cluster_addr":"172.21.15.203:6819/110630","heartbeat_back_addr":"172.21.15.203:6823/110630","heartbeat_front_addr":"172.21.15.203:6821/110630","state":["exists","up"]},{"osd":3,"uuid":"3451afca-3f67-4a33-88bc-9c4a3fa71b37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":110642},{"type":"v1","addr":"172.21.15.203:6801","nonce":110642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":110642},{"type":"v1","addr":"172.21.15.203:6803","nonce":110642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":110642},{"type":"v1","addr":"172.21.15.203:6807","nonce":110642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":110642},{"type":"v1","addr":"172.21.15.203:6805","nonce":110642}]},"public_addr":"172.21.15.203:6801/110642","cluster_addr":"172.21.15.203:6803/110642","heartbeat_back_addr":"172.21.15.203:6807/110642","heartbeat_front_addr":"172.21.15.203:6805/110642","state":["exists","up"]},{"osd":4,"uuid":"53249305-7472-4032-88c6-b6f2eab2f939","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6845","nonce":110696},{"type":"v1","addr":"172.21.15.203:6847","nonce":110696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6849","nonce":110696},{"type":"v1","addr":"172.21.15.203:6851","nonce":110696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6854","nonce":110696},{"type":"v1","addr":"172.21.15.203:6855","nonce":110696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6852","nonce":110696},{"type":"v1","addr":"172.21.15.203:6853","nonce":110696}]},"public_addr":"172.21.15.203:6847/110696","cluster_addr":"172.21.15.203:6851/110696","heartbeat_back_addr":"172.21.15.203:6855/110696","heartbeat_front_addr":"172.21.15.203:6853/110696","state":["exists","up"]},{"osd":5,"uuid":"3c6cd644-1d97-4a74-815a-cbffa9685718","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":110644},{"type":"v1","addr":"172.21.15.203:6833","nonce":110644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":110644},{"type":"v1","addr":"172.21.15.203:6835","nonce":110644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":110644},{"type":"v1","addr":"172.21.15.203:6840","nonce":110644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":110644},{"type":"v1","addr":"172.21.15.203:6837","nonce":110644}]},"public_addr":"172.21.15.203:6833/110644","cluster_addr":"172.21.15.203:6835/110644","heartbeat_back_addr":"172.21.15.203:6840/110644","heartbeat_front_addr":"172.21.15.203:6837/110644","state":["exists","up"]},{"osd":6,"uuid":"1426af76-6197-499f-8aa3-9202c91b6fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6839","nonce":110685},{"type":"v1","addr":"172.21.15.203:6841","nonce":110685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6842","nonce":110685},{"type":"v1","addr":"172.21.15.203:6843","nonce":110685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6848","nonce":110685},{"type":"v1","addr":"172.21.15.203:6850","nonce":110685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6844","nonce":110685},{"type":"v1","addr":"172.21.15.203:6846","nonce":110685}]},"public_addr":"172.21.15.203:6841/110685","cluster_addr":"172.21.15.203:6843/110685","heartbeat_back_addr":"172.21.15.203:6850/110685","heartbeat_front_addr":"172.21.15.203:6846/110685","state":["exists","up"]},{"osd":7,"uuid":"a13fb8e1-a46d-4398-956d-ce4d7b04da48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6859","nonce":110783},{"type":"v1","addr":"172.21.15.203:6861","nonce":110783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6864","nonce":110783},{"type":"v1","addr":"172.21.15.203:6866","nonce":110783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6870","nonce":110783},{"type":"v1","addr":"172.21.15.203:6871","nonce":110783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6868","nonce":110783},{"type":"v1","addr":"172.21.15.203:6869","nonce":110783}]},"public_addr":"172.21.15.203:6861/110783","cluster_addr":"172.21.15.203:6866/110783","heartbeat_back_addr":"172.21.15.203:6871/110783","heartbeat_front_addr":"172.21.15.203:6869/110783","state":["exists","up"]},{"osd":8,"uuid":"cc46f5f4-d476-4ff6-b7f9-537263cb24cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6856","nonce":110781},{"type":"v1","addr":"172.21.15.203:6857","nonce":110781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6858","nonce":110781},{"type":"v1","addr":"172.21.15.203:6860","nonce":110781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6865","nonce":110781},{"type":"v1","addr":"172.21.15.203:6867","nonce":110781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6862","nonce":110781},{"type":"v1","addr":"172.21.15.203:6863","nonce":110781}]},"public_addr":"172.21.15.203:6857/110781","cluster_addr":"172.21.15.203:6860/110781","heartbeat_back_addr":"172.21.15.203:6867/110781","heartbeat_front_addr":"172.21.15.203:6863/110781","state":["exists","up"]},{"osd":9,"uuid":"2ac16a61-4dcd-43be-a686-f9959a8901ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6872","nonce":110792},{"type":"v1","addr":"172.21.15.203:6873","nonce":110792}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6874","nonce":110792},{"type":"v1","addr":"172.21.15.203:6875","nonce":110792}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6878","nonce":110792},{"type":"v1","addr":"172.21.15.203:6879","nonce":110792}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6876","nonce":110792},{"type":"v1","addr":"172.21.15.203:6877","nonce":110792}]},"public_addr":"172.21.15.203:6873/110792","cluster_addr":"172.21.15.203:6875/110792","heartbeat_back_addr":"172.21.15.203:6879/110792","heartbeat_front_addr":"172.21.15.203:6877/110792","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:27.924170+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.162198+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.104251+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:27.901694+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.209516+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.128175+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.144209+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.228683+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.195999+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.258473+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-25T05:42:35.418 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-01-25T05:42:35.418 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-01-25T05:42:35.419 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-01-25T05:42:35.419 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-01-25T05:42:35.420 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-01-25T05:42:35.420 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-01-25T05:42:35.421 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-01-25T05:42:35.422 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-01-25T05:42:35.422 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2023-01-25T05:42:35.423 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2023-01-25T05:42:35.754 INFO:teuthology.orchestra.run.smithi203.stdout:64424509443 2023-01-25T05:42:35.755 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-01-25T05:42:35.763 INFO:teuthology.orchestra.run.smithi203.stdout:64424509443 2023-01-25T05:42:35.763 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2023-01-25T05:42:35.764 INFO:teuthology.orchestra.run.smithi203.stdout:64424509443 2023-01-25T05:42:35.765 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-01-25T05:42:35.774 INFO:teuthology.orchestra.run.smithi203.stdout:64424509444 2023-01-25T05:42:35.775 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-01-25T05:42:35.782 INFO:teuthology.orchestra.run.smithi203.stdout:64424509443 2023-01-25T05:42:35.783 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2023-01-25T05:42:35.802 INFO:teuthology.orchestra.run.smithi203.stdout:64424509443 2023-01-25T05:42:35.803 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-01-25T05:42:35.810 INFO:teuthology.orchestra.run.smithi203.stdout:64424509443 2023-01-25T05:42:35.811 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-01-25T05:42:35.818 INFO:teuthology.orchestra.run.smithi203.stdout:64424509443 2023-01-25T05:42:35.818 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-01-25T05:42:35.827 INFO:teuthology.orchestra.run.smithi203.stdout:64424509444 2023-01-25T05:42:35.827 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-01-25T05:42:35.832 INFO:teuthology.orchestra.run.smithi203.stdout:64424509444 2023-01-25T05:42:35.833 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-01-25T05:42:36.490 INFO:teuthology.orchestra.run.smithi203.stdout:64424509443 2023-01-25T05:42:36.500 INFO:teuthology.orchestra.run.smithi203.stdout:64424509443 2023-01-25T05:42:36.515 INFO:teuthology.orchestra.run.smithi203.stdout:64424509443 2023-01-25T05:42:36.518 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.9 2023-01-25T05:42:36.518 DEBUG:teuthology.parallel:result is None 2023-01-25T05:42:36.537 INFO:teuthology.orchestra.run.smithi203.stdout:64424509444 2023-01-25T05:42:36.538 INFO:teuthology.orchestra.run.smithi203.stdout:64424509443 2023-01-25T05:42:36.544 INFO:teuthology.orchestra.run.smithi203.stdout:64424509443 2023-01-25T05:42:36.550 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.8 2023-01-25T05:42:36.550 DEBUG:teuthology.parallel:result is None 2023-01-25T05:42:36.564 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.6 2023-01-25T05:42:36.564 DEBUG:teuthology.parallel:result is None 2023-01-25T05:42:36.569 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.3 2023-01-25T05:42:36.569 DEBUG:teuthology.parallel:result is None 2023-01-25T05:42:36.577 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.2 2023-01-25T05:42:36.578 DEBUG:teuthology.parallel:result is None 2023-01-25T05:42:36.581 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.1 2023-01-25T05:42:36.582 DEBUG:teuthology.parallel:result is None 2023-01-25T05:42:36.584 INFO:teuthology.orchestra.run.smithi203.stdout:64424509443 2023-01-25T05:42:36.598 INFO:teuthology.orchestra.run.smithi203.stdout:64424509444 2023-01-25T05:42:36.603 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.0 2023-01-25T05:42:36.603 DEBUG:teuthology.parallel:result is None 2023-01-25T05:42:36.614 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.7 2023-01-25T05:42:36.614 DEBUG:teuthology.parallel:result is None 2023-01-25T05:42:36.637 INFO:teuthology.orchestra.run.smithi203.stdout:64424509443 2023-01-25T05:42:36.652 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.4 2023-01-25T05:42:36.652 DEBUG:teuthology.parallel:result is None 2023-01-25T05:42:36.663 INFO:teuthology.orchestra.run.smithi203.stdout:64424509444 2023-01-25T05:42:36.678 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.5 2023-01-25T05:42:36.678 DEBUG:teuthology.parallel:result is None 2023-01-25T05:42:36.678 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-01-25T05:42:36.679 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-25T05:42:37.030 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:42:37.032 INFO:teuthology.orchestra.run.smithi203.stderr:dumped all 2023-01-25T05:42:37.043 INFO:teuthology.orchestra.run.smithi203.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-01-25T05:42:36.092062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":9,"kb":1048576000,"kb_used":51040,"kb_used_data":1680,"kb_used_omap":0,"kb_used_meta":49280,"kb_avail":1048524960,"statfs":{"total":1073741824000,"available":1073689559040,"internally_reserved":0,"allocated":1720320,"data_stored":533230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50462720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-01-25T05:42:34.471304+0000","last_change":"2023-01-25T05:42:34.470834+0000","last_active":"2023-01-25T05:42:34.471304+0000","last_peered":"2023-01-25T05:42:34.471304+0000","last_clean":"2023-01-25T05:42:34.471304+0000","last_became_active":"2023-01-25T05:42:34.470559+0000","last_became_peered":"2023-01-25T05:42:34.470559+0000","last_unstale":"2023-01-25T05:42:34.471304+0000","last_undegraded":"2023-01-25T05:42:34.471304+0000","last_fullsized":"2023-01-25T05:42:34.471304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T11:46:24.713259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-01-25T05:42:34.471309+0000","last_change":"2023-01-25T05:42:34.470900+0000","last_active":"2023-01-25T05:42:34.471309+0000","last_peered":"2023-01-25T05:42:34.471309+0000","last_clean":"2023-01-25T05:42:34.471309+0000","last_became_active":"2023-01-25T05:42:34.470583+0000","last_became_peered":"2023-01-25T05:42:34.470583+0000","last_unstale":"2023-01-25T05:42:34.471309+0000","last_undegraded":"2023-01-25T05:42:34.471309+0000","last_fullsized":"2023-01-25T05:42:34.471309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T10:25:21.673953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-01-25T05:42:34.358673+0000","last_change":"2023-01-25T05:42:34.357817+0000","last_active":"2023-01-25T05:42:34.358673+0000","last_peered":"2023-01-25T05:42:34.358673+0000","last_clean":"2023-01-25T05:42:34.358673+0000","last_became_active":"2023-01-25T05:42:34.357524+0000","last_became_peered":"2023-01-25T05:42:34.357524+0000","last_unstale":"2023-01-25T05:42:34.358673+0000","last_undegraded":"2023-01-25T05:42:34.358673+0000","last_fullsized":"2023-01-25T05:42:34.358673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T08:35:57.204848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-01-25T05:42:34.359124+0000","last_change":"2023-01-25T05:42:34.359124+0000","last_active":"2023-01-25T05:42:34.359124+0000","last_peered":"2023-01-25T05:42:34.359124+0000","last_clean":"2023-01-25T05:42:34.359124+0000","last_became_active":"2023-01-25T05:42:34.358883+0000","last_became_peered":"2023-01-25T05:42:34.358883+0000","last_unstale":"2023-01-25T05:42:34.359124+0000","last_undegraded":"2023-01-25T05:42:34.359124+0000","last_fullsized":"2023-01-25T05:42:34.359124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T06:47:27.530116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-01-25T05:42:34.470965+0000","last_change":"2023-01-25T05:42:34.470965+0000","last_active":"2023-01-25T05:42:34.470965+0000","last_peered":"2023-01-25T05:42:34.470965+0000","last_clean":"2023-01-25T05:42:34.470965+0000","last_became_active":"2023-01-25T05:42:34.470725+0000","last_became_peered":"2023-01-25T05:42:34.470725+0000","last_unstale":"2023-01-25T05:42:34.470965+0000","last_undegraded":"2023-01-25T05:42:34.470965+0000","last_fullsized":"2023-01-25T05:42:34.470965+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T08:35:57.204848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-01-25T05:42:34.437520+0000","last_change":"2023-01-25T05:42:34.437520+0000","last_active":"2023-01-25T05:42:34.437520+0000","last_peered":"2023-01-25T05:42:34.437520+0000","last_clean":"2023-01-25T05:42:34.437520+0000","last_became_active":"2023-01-25T05:42:34.437276+0000","last_became_peered":"2023-01-25T05:42:34.437276+0000","last_unstale":"2023-01-25T05:42:34.437520+0000","last_undegraded":"2023-01-25T05:42:34.437520+0000","last_fullsized":"2023-01-25T05:42:34.437520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T10:25:21.673953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-01-25T05:42:34.359579+0000","last_change":"2023-01-25T05:42:34.359208+0000","last_active":"2023-01-25T05:42:34.359579+0000","last_peered":"2023-01-25T05:42:34.359579+0000","last_clean":"2023-01-25T05:42:34.359579+0000","last_became_active":"2023-01-25T05:42:34.358950+0000","last_became_peered":"2023-01-25T05:42:34.358950+0000","last_unstale":"2023-01-25T05:42:34.359579+0000","last_undegraded":"2023-01-25T05:42:34.359579+0000","last_fullsized":"2023-01-25T05:42:34.359579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T08:35:57.204848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-01-25T05:42:34.359382+0000","last_change":"2023-01-25T05:42:34.359382+0000","last_active":"2023-01-25T05:42:34.359382+0000","last_peered":"2023-01-25T05:42:34.359382+0000","last_clean":"2023-01-25T05:42:34.359382+0000","last_became_active":"2023-01-25T05:42:34.359139+0000","last_became_peered":"2023-01-25T05:42:34.359139+0000","last_unstale":"2023-01-25T05:42:34.359382+0000","last_undegraded":"2023-01-25T05:42:34.359382+0000","last_fullsized":"2023-01-25T05:42:34.359382+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T11:46:24.713259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8],"acting":[4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":15,"seq":64424509443,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5104,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":104852496,"statfs":{"total":107374182400,"available":107368955904,"internally_reserved":0,"allocated":172032,"data_stored":53323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]}]},{"osd":8,"up_from":15,"seq":64424509443,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5104,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":104852496,"statfs":{"total":107374182400,"available":107368955904,"internally_reserved":0,"allocated":172032,"data_stored":53323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]}]},{"osd":7,"up_from":15,"seq":64424509444,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5104,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":104852496,"statfs":{"total":107374182400,"available":107368955904,"internally_reserved":0,"allocated":172032,"data_stored":53323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]}]},{"osd":6,"up_from":15,"seq":64424509443,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5104,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":104852496,"statfs":{"total":107374182400,"available":107368955904,"internally_reserved":0,"allocated":172032,"data_stored":53323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]}]},{"osd":1,"up_from":15,"seq":64424509444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5104,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":104852496,"statfs":{"total":107374182400,"available":107368955904,"internally_reserved":0,"allocated":172032,"data_stored":53323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509443,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5104,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":104852496,"statfs":{"total":107374182400,"available":107368955904,"internally_reserved":0,"allocated":172032,"data_stored":53323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]},{"osd":2,"up_from":15,"seq":64424509443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":5104,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":104852496,"statfs":{"total":107374182400,"available":107368955904,"internally_reserved":0,"allocated":172032,"data_stored":53323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]}]},{"osd":3,"up_from":15,"seq":64424509443,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5104,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":104852496,"statfs":{"total":107374182400,"available":107368955904,"internally_reserved":0,"allocated":172032,"data_stored":53323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509443,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5104,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":104852496,"statfs":{"total":107374182400,"available":107368955904,"internally_reserved":0,"allocated":172032,"data_stored":53323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]}]},{"osd":5,"up_from":15,"seq":64424509444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5104,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":104852496,"statfs":{"total":107374182400,"available":107368955904,"internally_reserved":0,"allocated":172032,"data_stored":53323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-25T05:42:37.045 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-25T05:42:37.360 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:42:37.362 INFO:teuthology.orchestra.run.smithi203.stderr:dumped all 2023-01-25T05:42:37.373 INFO:teuthology.orchestra.run.smithi203.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-01-25T05:42:36.092062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":9,"kb":1048576000,"kb_used":51040,"kb_used_data":1680,"kb_used_omap":0,"kb_used_meta":49280,"kb_avail":1048524960,"statfs":{"total":1073741824000,"available":1073689559040,"internally_reserved":0,"allocated":1720320,"data_stored":533230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50462720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-01-25T05:42:34.471304+0000","last_change":"2023-01-25T05:42:34.470834+0000","last_active":"2023-01-25T05:42:34.471304+0000","last_peered":"2023-01-25T05:42:34.471304+0000","last_clean":"2023-01-25T05:42:34.471304+0000","last_became_active":"2023-01-25T05:42:34.470559+0000","last_became_peered":"2023-01-25T05:42:34.470559+0000","last_unstale":"2023-01-25T05:42:34.471304+0000","last_undegraded":"2023-01-25T05:42:34.471304+0000","last_fullsized":"2023-01-25T05:42:34.471304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T11:46:24.713259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-01-25T05:42:34.471309+0000","last_change":"2023-01-25T05:42:34.470900+0000","last_active":"2023-01-25T05:42:34.471309+0000","last_peered":"2023-01-25T05:42:34.471309+0000","last_clean":"2023-01-25T05:42:34.471309+0000","last_became_active":"2023-01-25T05:42:34.470583+0000","last_became_peered":"2023-01-25T05:42:34.470583+0000","last_unstale":"2023-01-25T05:42:34.471309+0000","last_undegraded":"2023-01-25T05:42:34.471309+0000","last_fullsized":"2023-01-25T05:42:34.471309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T10:25:21.673953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9],"acting":[6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-01-25T05:42:34.358673+0000","last_change":"2023-01-25T05:42:34.357817+0000","last_active":"2023-01-25T05:42:34.358673+0000","last_peered":"2023-01-25T05:42:34.358673+0000","last_clean":"2023-01-25T05:42:34.358673+0000","last_became_active":"2023-01-25T05:42:34.357524+0000","last_became_peered":"2023-01-25T05:42:34.357524+0000","last_unstale":"2023-01-25T05:42:34.358673+0000","last_undegraded":"2023-01-25T05:42:34.358673+0000","last_fullsized":"2023-01-25T05:42:34.358673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T08:35:57.204848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8],"acting":[5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-01-25T05:42:34.359124+0000","last_change":"2023-01-25T05:42:34.359124+0000","last_active":"2023-01-25T05:42:34.359124+0000","last_peered":"2023-01-25T05:42:34.359124+0000","last_clean":"2023-01-25T05:42:34.359124+0000","last_became_active":"2023-01-25T05:42:34.358883+0000","last_became_peered":"2023-01-25T05:42:34.358883+0000","last_unstale":"2023-01-25T05:42:34.359124+0000","last_undegraded":"2023-01-25T05:42:34.359124+0000","last_fullsized":"2023-01-25T05:42:34.359124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T06:47:27.530116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0],"acting":[8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-01-25T05:42:34.470965+0000","last_change":"2023-01-25T05:42:34.470965+0000","last_active":"2023-01-25T05:42:34.470965+0000","last_peered":"2023-01-25T05:42:34.470965+0000","last_clean":"2023-01-25T05:42:34.470965+0000","last_became_active":"2023-01-25T05:42:34.470725+0000","last_became_peered":"2023-01-25T05:42:34.470725+0000","last_unstale":"2023-01-25T05:42:34.470965+0000","last_undegraded":"2023-01-25T05:42:34.470965+0000","last_fullsized":"2023-01-25T05:42:34.470965+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T08:35:57.204848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-01-25T05:42:34.437520+0000","last_change":"2023-01-25T05:42:34.437520+0000","last_active":"2023-01-25T05:42:34.437520+0000","last_peered":"2023-01-25T05:42:34.437520+0000","last_clean":"2023-01-25T05:42:34.437520+0000","last_became_active":"2023-01-25T05:42:34.437276+0000","last_became_peered":"2023-01-25T05:42:34.437276+0000","last_unstale":"2023-01-25T05:42:34.437520+0000","last_undegraded":"2023-01-25T05:42:34.437520+0000","last_fullsized":"2023-01-25T05:42:34.437520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T10:25:21.673953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-01-25T05:42:34.359579+0000","last_change":"2023-01-25T05:42:34.359208+0000","last_active":"2023-01-25T05:42:34.359579+0000","last_peered":"2023-01-25T05:42:34.359579+0000","last_clean":"2023-01-25T05:42:34.359579+0000","last_became_active":"2023-01-25T05:42:34.358950+0000","last_became_peered":"2023-01-25T05:42:34.358950+0000","last_unstale":"2023-01-25T05:42:34.359579+0000","last_undegraded":"2023-01-25T05:42:34.359579+0000","last_fullsized":"2023-01-25T05:42:34.359579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T08:35:57.204848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-01-25T05:42:34.359382+0000","last_change":"2023-01-25T05:42:34.359382+0000","last_active":"2023-01-25T05:42:34.359382+0000","last_peered":"2023-01-25T05:42:34.359382+0000","last_clean":"2023-01-25T05:42:34.359382+0000","last_became_active":"2023-01-25T05:42:34.359139+0000","last_became_peered":"2023-01-25T05:42:34.359139+0000","last_unstale":"2023-01-25T05:42:34.359382+0000","last_undegraded":"2023-01-25T05:42:34.359382+0000","last_fullsized":"2023-01-25T05:42:34.359382+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T11:46:24.713259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8],"acting":[4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":15,"seq":64424509443,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5104,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":104852496,"statfs":{"total":107374182400,"available":107368955904,"internally_reserved":0,"allocated":172032,"data_stored":53323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]}]},{"osd":8,"up_from":15,"seq":64424509443,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5104,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":104852496,"statfs":{"total":107374182400,"available":107368955904,"internally_reserved":0,"allocated":172032,"data_stored":53323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]}]},{"osd":7,"up_from":15,"seq":64424509444,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5104,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":104852496,"statfs":{"total":107374182400,"available":107368955904,"internally_reserved":0,"allocated":172032,"data_stored":53323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]}]},{"osd":6,"up_from":15,"seq":64424509443,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5104,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":104852496,"statfs":{"total":107374182400,"available":107368955904,"internally_reserved":0,"allocated":172032,"data_stored":53323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55600000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]}]},{"osd":1,"up_from":15,"seq":64424509444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5104,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":104852496,"statfs":{"total":107374182400,"available":107368955904,"internally_reserved":0,"allocated":172032,"data_stored":53323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509443,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5104,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":104852496,"statfs":{"total":107374182400,"available":107368955904,"internally_reserved":0,"allocated":172032,"data_stored":53323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]},{"osd":2,"up_from":15,"seq":64424509443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":5104,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":104852496,"statfs":{"total":107374182400,"available":107368955904,"internally_reserved":0,"allocated":172032,"data_stored":53323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.625}]}]},{"osd":3,"up_from":15,"seq":64424509443,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5104,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":104852496,"statfs":{"total":107374182400,"available":107368955904,"internally_reserved":0,"allocated":172032,"data_stored":53323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509443,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5104,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":104852496,"statfs":{"total":107374182400,"available":107368955904,"internally_reserved":0,"allocated":172032,"data_stored":53323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]}]},{"osd":5,"up_from":15,"seq":64424509444,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5104,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":104852496,"statfs":{"total":107374182400,"available":107368955904,"internally_reserved":0,"allocated":172032,"data_stored":53323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-25T05:42:37.375 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-01-25T05:42:37.375 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-01-25T05:42:37.375 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-01-25T05:42:37.376 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-01-25T05:42:37.725 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:42:37.726 INFO:teuthology.orchestra.run.smithi203.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-01-25T05:42:37.739 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-01-25T05:42:37.740 INFO:teuthology.run_tasks:Running task workunit... 2023-01-25T05:42:37.753 INFO:tasks.workunit:Pulling workunits from ref 61996d73ab738798e4ee0055c86457f5c459da4a 2023-01-25T05:42:37.754 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-01-25T05:42:37.754 INFO:tasks.workunit:timeout=3h 2023-01-25T05:42:37.754 INFO:tasks.workunit:cleanup=True 2023-01-25T05:42:37.755 DEBUG:teuthology.orchestra.run.smithi203:> stat -- /home/ubuntu/cephtest/mnt.0 2023-01-25T05:42:37.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-25T05:42:37.773 INFO:teuthology.orchestra.run.smithi203.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-01-25T05:42:37.774 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-01-25T05:42:37.832 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-01-25T05:42:37.832 DEBUG:teuthology.orchestra.run.smithi203:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-01-25T05:42:37.890 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 61996d73ab738798e4ee0055c86457f5c459da4a 2023-01-25T05:42:37.949 INFO:tasks.workunit.client.0.smithi203.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-01-25T05:43:17.720 INFO:tasks.workunit.client.0.smithi203.stderr:Note: switching to '61996d73ab738798e4ee0055c86457f5c459da4a'. 2023-01-25T05:43:17.720 INFO:tasks.workunit.client.0.smithi203.stderr: 2023-01-25T05:43:17.720 INFO:tasks.workunit.client.0.smithi203.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-01-25T05:43:17.721 INFO:tasks.workunit.client.0.smithi203.stderr:changes and commit them, and you can discard any commits you make in this 2023-01-25T05:43:17.721 INFO:tasks.workunit.client.0.smithi203.stderr:state without impacting any branches by switching back to a branch. 2023-01-25T05:43:17.721 INFO:tasks.workunit.client.0.smithi203.stderr: 2023-01-25T05:43:17.721 INFO:tasks.workunit.client.0.smithi203.stderr:If you want to create a new branch to retain commits you create, you may 2023-01-25T05:43:17.721 INFO:tasks.workunit.client.0.smithi203.stderr:do so (now or later) by using -c with the switch command. Example: 2023-01-25T05:43:17.722 INFO:tasks.workunit.client.0.smithi203.stderr: 2023-01-25T05:43:17.722 INFO:tasks.workunit.client.0.smithi203.stderr: git switch -c 2023-01-25T05:43:17.722 INFO:tasks.workunit.client.0.smithi203.stderr: 2023-01-25T05:43:17.722 INFO:tasks.workunit.client.0.smithi203.stderr:Or undo this operation with: 2023-01-25T05:43:17.722 INFO:tasks.workunit.client.0.smithi203.stderr: 2023-01-25T05:43:17.723 INFO:tasks.workunit.client.0.smithi203.stderr: git switch - 2023-01-25T05:43:17.723 INFO:tasks.workunit.client.0.smithi203.stderr: 2023-01-25T05:43:17.723 INFO:tasks.workunit.client.0.smithi203.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-01-25T05:43:17.723 INFO:tasks.workunit.client.0.smithi203.stderr: 2023-01-25T05:43:17.724 INFO:tasks.workunit.client.0.smithi203.stderr:HEAD is now at 61996d73ab7 Merge branch 'wip-tserong-qa-crash-chown' of https://github.com/SUSE/ceph into wip-yuri2-testing-2023-01-23-0928 2023-01-25T05:43:17.725 DEBUG:teuthology.orchestra.run.smithi203:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-01-25T05:43:17.792 INFO:tasks.workunit.client.0.smithi203.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-01-25T05:43:17.792 INFO:tasks.workunit.client.0.smithi203.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-01-25T05:43:17.793 INFO:tasks.workunit.client.0.smithi203.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-01-25T05:43:18.344 INFO:tasks.workunit.client.0.smithi203.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-01-25T05:43:18.412 INFO:tasks.workunit.client.0.smithi203.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-01-25T05:43:18.449 INFO:tasks.workunit.client.0.smithi203.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-01-25T05:43:18.451 INFO:tasks.workunit.client.0.smithi203.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-01-25T05:43:18.451 INFO:tasks.workunit.client.0.smithi203.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-01-25T05:43:18.488 INFO:tasks.workunit.client.0.smithi203.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-01-25T05:43:18.491 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-01-25T05:43:18.491 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-01-25T05:43:18.551 INFO:tasks.workunit:Running workunits matching rados/test_health_warnings.sh on client.0... 2023-01-25T05:43:18.553 INFO:tasks.workunit:Running workunit rados/test_health_warnings.sh... 2023-01-25T05:43:18.554 DEBUG:teuthology.orchestra.run.smithi203:workunit test rados/test_health_warnings.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=61996d73ab738798e4ee0055c86457f5c459da4a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_health_warnings.sh 2023-01-25T05:43:18.613 INFO:tasks.workunit.client.0.smithi203.stderr:+ crushtool -o crushmap --build --num_osds 10 host straw 2 rack straw 2 row straw 2 root straw 0 2023-01-25T05:43:18.629 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph osd setcrushmap -i crushmap 2023-01-25T05:43:20.439 INFO:tasks.workunit.client.0.smithi203.stderr:6 2023-01-25T05:43:20.453 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph osd tree 2023-01-25T05:43:20.865 INFO:tasks.workunit.client.0.smithi203.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2023-01-25T05:43:20.865 INFO:tasks.workunit.client.0.smithi203.stdout:-11 10.00000 root root 2023-01-25T05:43:20.866 INFO:tasks.workunit.client.0.smithi203.stdout: -9 8.00000 row row0 2023-01-25T05:43:20.866 INFO:tasks.workunit.client.0.smithi203.stdout: -6 4.00000 rack rack0 2023-01-25T05:43:20.866 INFO:tasks.workunit.client.0.smithi203.stdout: -1 2.00000 host host0 2023-01-25T05:43:20.866 INFO:tasks.workunit.client.0.smithi203.stdout: 0 1.00000 osd.0 up 1.00000 1.00000 2023-01-25T05:43:20.866 INFO:tasks.workunit.client.0.smithi203.stdout: 1 1.00000 osd.1 up 1.00000 1.00000 2023-01-25T05:43:20.867 INFO:tasks.workunit.client.0.smithi203.stdout: -2 2.00000 host host1 2023-01-25T05:43:20.867 INFO:tasks.workunit.client.0.smithi203.stdout: 2 1.00000 osd.2 up 1.00000 1.00000 2023-01-25T05:43:20.867 INFO:tasks.workunit.client.0.smithi203.stdout: 3 1.00000 osd.3 up 1.00000 1.00000 2023-01-25T05:43:20.867 INFO:tasks.workunit.client.0.smithi203.stdout: -7 4.00000 rack rack1 2023-01-25T05:43:20.867 INFO:tasks.workunit.client.0.smithi203.stdout: -3 2.00000 host host2 2023-01-25T05:43:20.868 INFO:tasks.workunit.client.0.smithi203.stdout: 4 1.00000 osd.4 up 1.00000 1.00000 2023-01-25T05:43:20.868 INFO:tasks.workunit.client.0.smithi203.stdout: 5 1.00000 osd.5 up 1.00000 1.00000 2023-01-25T05:43:20.868 INFO:tasks.workunit.client.0.smithi203.stdout: -4 2.00000 host host3 2023-01-25T05:43:20.868 INFO:tasks.workunit.client.0.smithi203.stdout: 6 1.00000 osd.6 up 1.00000 1.00000 2023-01-25T05:43:20.868 INFO:tasks.workunit.client.0.smithi203.stdout: 7 1.00000 osd.7 up 1.00000 1.00000 2023-01-25T05:43:20.869 INFO:tasks.workunit.client.0.smithi203.stdout:-10 2.00000 row row1 2023-01-25T05:43:20.869 INFO:tasks.workunit.client.0.smithi203.stdout: -8 2.00000 rack rack2 2023-01-25T05:43:20.870 INFO:tasks.workunit.client.0.smithi203.stdout: -5 2.00000 host host4 2023-01-25T05:43:20.870 INFO:tasks.workunit.client.0.smithi203.stdout: 8 1.00000 osd.8 up 1.00000 1.00000 2023-01-25T05:43:20.870 INFO:tasks.workunit.client.0.smithi203.stdout: 9 1.00000 osd.9 up 1.00000 1.00000 2023-01-25T05:43:20.877 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph tell 'osd.*' injectargs --osd_max_markdown_count 1024 --osd_max_markdown_period 1 2023-01-25T05:43:21.012 INFO:tasks.workunit.client.0.smithi203.stdout:osd.0: {} 2023-01-25T05:43:21.046 INFO:tasks.workunit.client.0.smithi203.stdout:osd.1: {} 2023-01-25T05:43:21.078 INFO:tasks.workunit.client.0.smithi203.stdout:osd.2: {} 2023-01-25T05:43:21.110 INFO:tasks.workunit.client.0.smithi203.stdout:osd.3: {} 2023-01-25T05:43:21.143 INFO:tasks.workunit.client.0.smithi203.stdout:osd.4: {} 2023-01-25T05:43:21.175 INFO:tasks.workunit.client.0.smithi203.stdout:osd.5: {} 2023-01-25T05:43:21.207 INFO:tasks.workunit.client.0.smithi203.stdout:osd.6: {} 2023-01-25T05:43:21.239 INFO:tasks.workunit.client.0.smithi203.stdout:osd.7: {} 2023-01-25T05:43:21.271 INFO:tasks.workunit.client.0.smithi203.stdout:osd.8: {} 2023-01-25T05:43:21.303 INFO:tasks.workunit.client.0.smithi203.stdout:osd.9: {} 2023-01-25T05:43:21.314 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph osd set noout 2023-01-25T05:43:23.435 INFO:tasks.workunit.client.0.smithi203.stderr:noout is set 2023-01-25T05:43:23.447 INFO:tasks.workunit.client.0.smithi203.stderr:+ test_mark_two_osds_same_host_down 2023-01-25T05:43:23.447 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph osd set noup 2023-01-25T05:43:25.442 INFO:tasks.workunit.client.0.smithi203.stderr:noup is set 2023-01-25T05:43:25.452 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph osd down osd.0 osd.1 2023-01-25T05:43:25.754 INFO:tasks.ceph.mon.a.smithi203.stderr:2023-01-25T05:43:25.752+0000 7f0213607700 -1 mon.a@0(leader).osd e25 definitely_dead 0 2023-01-25T05:43:26.584 INFO:tasks.ceph.mon.a.smithi203.stderr:2023-01-25T05:43:26.582+0000 7f0213607700 -1 mon.a@0(leader).osd e26 definitely_dead 0 2023-01-25T05:43:26.587 INFO:tasks.workunit.client.0.smithi203.stderr:osd.0 is already down. osd.1 is already down. 2023-01-25T05:43:26.596 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health detail 2023-01-25T05:43:27.155 INFO:tasks.workunit.client.0.smithi203.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2023-01-25T05:43:27.156 INFO:tasks.workunit.client.0.smithi203.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2023-01-25T05:43:27.156 INFO:tasks.workunit.client.0.smithi203.stdout:[WRN] OSD_DOWN: 2 osds down 2023-01-25T05:43:27.156 INFO:tasks.workunit.client.0.smithi203.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2023-01-25T05:43:27.156 INFO:tasks.workunit.client.0.smithi203.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2023-01-25T05:43:27.156 INFO:tasks.workunit.client.0.smithi203.stdout:[WRN] OSD_HOST_DOWN: 1 host (2 osds) down 2023-01-25T05:43:27.157 INFO:tasks.workunit.client.0.smithi203.stdout: host host0 (root=root,row=row0,rack=rack0) (2 osds) is down 2023-01-25T05:43:27.168 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health 2023-01-25T05:43:27.168 INFO:tasks.workunit.client.0.smithi203.stderr:+ grep '1 host' 2023-01-25T05:43:27.322 INFO:tasks.ceph.osd.0.smithi203.stderr:2023-01-25T05:43:27.319+0000 7f23cde5b700 -1 osd.0 26 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:27.448 INFO:tasks.ceph.osd.0.smithi203.stderr:2023-01-25T05:43:27.447+0000 7f23bfa29700 -1 osd.0 27 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:27.679 INFO:tasks.workunit.client.0.smithi203.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2023-01-25T05:43:27.680 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health 2023-01-25T05:43:27.680 INFO:tasks.workunit.client.0.smithi203.stderr:+ grep '2 osds' 2023-01-25T05:43:28.182 INFO:tasks.workunit.client.0.smithi203.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2023-01-25T05:43:28.183 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health detail 2023-01-25T05:43:28.183 INFO:tasks.workunit.client.0.smithi203.stderr:+ grep osd.0 2023-01-25T05:43:28.454 INFO:tasks.ceph.osd.0.smithi203.stderr:2023-01-25T05:43:28.452+0000 7f23bfa29700 -1 osd.0 28 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:28.758 INFO:tasks.workunit.client.0.smithi203.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2023-01-25T05:43:28.759 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health detail 2023-01-25T05:43:28.759 INFO:tasks.workunit.client.0.smithi203.stderr:+ grep osd.1 2023-01-25T05:43:29.275 INFO:tasks.ceph.osd.1.smithi203.stderr:2023-01-25T05:43:29.273+0000 7f129e455700 -1 osd.1 28 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:29.323 INFO:tasks.workunit.client.0.smithi203.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2023-01-25T05:43:29.324 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph osd unset noup 2023-01-25T05:43:29.455 INFO:tasks.ceph.osd.1.smithi203.stderr:2023-01-25T05:43:29.452+0000 7f128f822700 -1 osd.1 29 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:29.455 INFO:tasks.ceph.osd.0.smithi203.stderr:2023-01-25T05:43:29.452+0000 7f23bfa29700 -1 osd.0 29 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:30.460 INFO:tasks.ceph.osd.0.smithi203.stderr:2023-01-25T05:43:30.458+0000 7f23c7a39700 -1 osd.0 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:43:30.461 INFO:tasks.ceph.osd.1.smithi203.stderr:2023-01-25T05:43:30.458+0000 7f1297832700 -1 osd.1 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:43:31.467 INFO:tasks.workunit.client.0.smithi203.stderr:noup is unset 2023-01-25T05:43:31.479 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait_for_healthy 2023-01-25T05:43:31.480 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health 2023-01-25T05:43:31.480 INFO:tasks.workunit.client.0.smithi203.stderr:+ grep down 2023-01-25T05:43:31.985 INFO:tasks.workunit.client.0.smithi203.stderr:+ test_mark_two_osds_same_rack_down 2023-01-25T05:43:31.986 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph osd set noup 2023-01-25T05:43:33.476 INFO:tasks.workunit.client.0.smithi203.stderr:noup is set 2023-01-25T05:43:33.488 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph osd down osd.8 osd.9 2023-01-25T05:43:33.795 INFO:tasks.ceph.mon.a.smithi203.stderr:2023-01-25T05:43:33.793+0000 7f0213607700 -1 mon.a@0(leader).osd e33 definitely_dead 0 2023-01-25T05:43:34.620 INFO:tasks.ceph.mon.a.smithi203.stderr:2023-01-25T05:43:34.618+0000 7f0213607700 -1 mon.a@0(leader).osd e34 definitely_dead 0 2023-01-25T05:43:34.623 INFO:tasks.workunit.client.0.smithi203.stderr:osd.8 is already down. osd.9 is already down. 2023-01-25T05:43:34.633 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health detail 2023-01-25T05:43:35.192 INFO:tasks.workunit.client.0.smithi203.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2023-01-25T05:43:35.192 INFO:tasks.workunit.client.0.smithi203.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2023-01-25T05:43:35.193 INFO:tasks.workunit.client.0.smithi203.stdout:[WRN] OSD_DOWN: 2 osds down 2023-01-25T05:43:35.193 INFO:tasks.workunit.client.0.smithi203.stdout: osd.8 (root=root,row=row1,rack=rack2,host=host4) is down 2023-01-25T05:43:35.193 INFO:tasks.workunit.client.0.smithi203.stdout: osd.9 (root=root,row=row1,rack=rack2,host=host4) is down 2023-01-25T05:43:35.193 INFO:tasks.workunit.client.0.smithi203.stdout:[WRN] OSD_HOST_DOWN: 1 host (2 osds) down 2023-01-25T05:43:35.194 INFO:tasks.workunit.client.0.smithi203.stdout: host host4 (root=root,row=row1,rack=rack2) (2 osds) is down 2023-01-25T05:43:35.194 INFO:tasks.workunit.client.0.smithi203.stdout:[WRN] OSD_RACK_DOWN: 1 rack (2 osds) down 2023-01-25T05:43:35.194 INFO:tasks.workunit.client.0.smithi203.stdout: rack rack2 (root=root,row=row1) (2 osds) is down 2023-01-25T05:43:35.194 INFO:tasks.workunit.client.0.smithi203.stdout:[WRN] OSD_ROW_DOWN: 1 row (2 osds) down 2023-01-25T05:43:35.194 INFO:tasks.workunit.client.0.smithi203.stdout: row row1 (root=root) (2 osds) is down 2023-01-25T05:43:35.208 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health 2023-01-25T05:43:35.208 INFO:tasks.workunit.client.0.smithi203.stderr:+ grep '1 host' 2023-01-25T05:43:35.703 INFO:tasks.workunit.client.0.smithi203.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2023-01-25T05:43:35.704 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health 2023-01-25T05:43:35.704 INFO:tasks.workunit.client.0.smithi203.stderr:+ grep '1 rack' 2023-01-25T05:43:36.206 INFO:tasks.workunit.client.0.smithi203.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2023-01-25T05:43:36.207 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health 2023-01-25T05:43:36.207 INFO:tasks.workunit.client.0.smithi203.stderr:+ grep '1 row' 2023-01-25T05:43:36.713 INFO:tasks.workunit.client.0.smithi203.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2023-01-25T05:43:36.714 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health 2023-01-25T05:43:36.714 INFO:tasks.workunit.client.0.smithi203.stderr:+ grep '2 osds' 2023-01-25T05:43:37.174 INFO:tasks.ceph.osd.9.smithi203.stderr:2023-01-25T05:43:37.172+0000 7fe31e252700 -1 osd.9 35 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:37.217 INFO:tasks.workunit.client.0.smithi203.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down; 1 rack (2 osds) down; 1 row (2 osds) down 2023-01-25T05:43:37.218 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health detail 2023-01-25T05:43:37.218 INFO:tasks.workunit.client.0.smithi203.stderr:+ grep osd.8 2023-01-25T05:43:37.489 INFO:tasks.ceph.osd.9.smithi203.stderr:2023-01-25T05:43:37.487+0000 7fe30f61f700 -1 osd.9 36 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:37.793 INFO:tasks.workunit.client.0.smithi203.stdout: osd.8 (root=root,row=row1,rack=rack2,host=host4) is down 2023-01-25T05:43:37.793 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health detail 2023-01-25T05:43:37.794 INFO:tasks.workunit.client.0.smithi203.stderr:+ grep osd.9 2023-01-25T05:43:37.912 INFO:tasks.ceph.osd.8.smithi203.stderr:2023-01-25T05:43:37.910+0000 7f75c323a700 -1 osd.8 36 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:38.353 INFO:tasks.workunit.client.0.smithi203.stdout: osd.9 (root=root,row=row1,rack=rack2,host=host4) is down 2023-01-25T05:43:38.354 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph osd unset noup 2023-01-25T05:43:38.492 INFO:tasks.ceph.osd.9.smithi203.stderr:2023-01-25T05:43:38.490+0000 7fe30f61f700 -1 osd.9 37 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:38.493 INFO:tasks.ceph.osd.8.smithi203.stderr:2023-01-25T05:43:38.490+0000 7f75b4607700 -1 osd.8 37 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:39.497 INFO:tasks.ceph.osd.9.smithi203.stderr:2023-01-25T05:43:39.495+0000 7fe31762f700 -1 osd.9 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:43:39.498 INFO:tasks.ceph.osd.8.smithi203.stderr:2023-01-25T05:43:39.495+0000 7f75bc617700 -1 osd.8 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:43:40.509 INFO:tasks.workunit.client.0.smithi203.stderr:noup is unset 2023-01-25T05:43:40.521 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait_for_healthy 2023-01-25T05:43:40.522 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health 2023-01-25T05:43:40.522 INFO:tasks.workunit.client.0.smithi203.stderr:+ grep down 2023-01-25T05:43:41.028 INFO:tasks.workunit.client.0.smithi203.stderr:+ test_mark_all_but_last_osds_down 2023-01-25T05:43:41.029 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph osd set noup 2023-01-25T05:43:42.520 INFO:tasks.workunit.client.0.smithi203.stderr:noup is set 2023-01-25T05:43:42.533 INFO:tasks.workunit.client.0.smithi203.stderr:++ ceph osd ls 2023-01-25T05:43:42.533 INFO:tasks.workunit.client.0.smithi203.stderr:++ sed '$d' 2023-01-25T05:43:42.961 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph osd down 0 1 2 3 4 5 6 7 8 2023-01-25T05:43:43.274 INFO:tasks.ceph.mon.a.smithi203.stderr:2023-01-25T05:43:43.273+0000 7f0213607700 -1 mon.a@0(leader).osd e41 definitely_dead 0 2023-01-25T05:43:43.647 INFO:tasks.ceph.mon.a.smithi203.stderr:2023-01-25T05:43:43.646+0000 7f0213607700 -1 mon.a@0(leader).osd e42 definitely_dead 0 2023-01-25T05:43:43.650 INFO:tasks.workunit.client.0.smithi203.stderr:osd.0 is already down. osd.1 is already down. osd.2 is already down. osd.3 is already down. osd.4 is already down. osd.5 is already down. osd.6 is already down. osd.7 is already down. osd.8 is already down. 2023-01-25T05:43:43.660 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health detail 2023-01-25T05:43:43.797 INFO:tasks.ceph.osd.4.smithi203.stderr:2023-01-25T05:43:43.796+0000 7f985e25c700 -1 osd.4 42 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:44.220 INFO:tasks.workunit.client.0.smithi203.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2023-01-25T05:43:44.220 INFO:tasks.workunit.client.0.smithi203.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2023-01-25T05:43:44.221 INFO:tasks.workunit.client.0.smithi203.stdout:[WRN] OSD_DOWN: 9 osds down 2023-01-25T05:43:44.221 INFO:tasks.workunit.client.0.smithi203.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2023-01-25T05:43:44.221 INFO:tasks.workunit.client.0.smithi203.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2023-01-25T05:43:44.221 INFO:tasks.workunit.client.0.smithi203.stdout: osd.2 (root=root,row=row0,rack=rack0,host=host1) is down 2023-01-25T05:43:44.221 INFO:tasks.workunit.client.0.smithi203.stdout: osd.3 (root=root,row=row0,rack=rack0,host=host1) is down 2023-01-25T05:43:44.222 INFO:tasks.workunit.client.0.smithi203.stdout: osd.4 (root=root,row=row0,rack=rack1,host=host2) is down 2023-01-25T05:43:44.222 INFO:tasks.workunit.client.0.smithi203.stdout: osd.5 (root=root,row=row0,rack=rack1,host=host2) is down 2023-01-25T05:43:44.222 INFO:tasks.workunit.client.0.smithi203.stdout: osd.6 (root=root,row=row0,rack=rack1,host=host3) is down 2023-01-25T05:43:44.222 INFO:tasks.workunit.client.0.smithi203.stdout: osd.7 (root=root,row=row0,rack=rack1,host=host3) is down 2023-01-25T05:43:44.222 INFO:tasks.workunit.client.0.smithi203.stdout: osd.8 (root=root,row=row1,rack=rack2,host=host4) is down 2023-01-25T05:43:44.223 INFO:tasks.workunit.client.0.smithi203.stdout:[WRN] OSD_HOST_DOWN: 4 hosts (8 osds) down 2023-01-25T05:43:44.223 INFO:tasks.workunit.client.0.smithi203.stdout: host host0 (root=root,row=row0,rack=rack0) (2 osds) is down 2023-01-25T05:43:44.223 INFO:tasks.workunit.client.0.smithi203.stdout: host host1 (root=root,row=row0,rack=rack0) (2 osds) is down 2023-01-25T05:43:44.223 INFO:tasks.workunit.client.0.smithi203.stdout: host host2 (root=root,row=row0,rack=rack1) (2 osds) is down 2023-01-25T05:43:44.223 INFO:tasks.workunit.client.0.smithi203.stdout: host host3 (root=root,row=row0,rack=rack1) (2 osds) is down 2023-01-25T05:43:44.224 INFO:tasks.workunit.client.0.smithi203.stdout:[WRN] OSD_RACK_DOWN: 2 racks (8 osds) down 2023-01-25T05:43:44.224 INFO:tasks.workunit.client.0.smithi203.stdout: rack rack0 (root=root,row=row0) (4 osds) is down 2023-01-25T05:43:44.224 INFO:tasks.workunit.client.0.smithi203.stdout: rack rack1 (root=root,row=row0) (4 osds) is down 2023-01-25T05:43:44.224 INFO:tasks.workunit.client.0.smithi203.stdout:[WRN] OSD_ROW_DOWN: 1 row (8 osds) down 2023-01-25T05:43:44.224 INFO:tasks.workunit.client.0.smithi203.stdout: row row0 (root=root) (8 osds) is down 2023-01-25T05:43:44.233 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health 2023-01-25T05:43:44.233 INFO:tasks.workunit.client.0.smithi203.stderr:+ grep '1 row' 2023-01-25T05:43:44.515 INFO:tasks.ceph.osd.4.smithi203.stderr:2023-01-25T05:43:44.514+0000 7f984f629700 -1 osd.4 43 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:44.740 INFO:tasks.workunit.client.0.smithi203.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2023-01-25T05:43:44.740 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health 2023-01-25T05:43:44.741 INFO:tasks.workunit.client.0.smithi203.stderr:+ grep '2 racks' 2023-01-25T05:43:44.983 INFO:tasks.ceph.osd.3.smithi203.stderr:2023-01-25T05:43:44.982+0000 7f056c913700 -1 osd.3 43 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:45.243 INFO:tasks.workunit.client.0.smithi203.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2023-01-25T05:43:45.243 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health 2023-01-25T05:43:45.243 INFO:tasks.workunit.client.0.smithi203.stderr:+ grep '4 hosts' 2023-01-25T05:43:45.254 INFO:tasks.ceph.osd.2.smithi203.stderr:2023-01-25T05:43:45.253+0000 7f10daaf3700 -1 osd.2 43 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:45.270 INFO:tasks.ceph.osd.0.smithi203.stderr:2023-01-25T05:43:45.269+0000 7f23cd65a700 -1 osd.0 43 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:45.327 INFO:tasks.ceph.osd.7.smithi203.stderr:2023-01-25T05:43:45.326+0000 7f3446a5f700 -1 osd.7 43 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:45.410 INFO:tasks.ceph.osd.5.smithi203.stderr:2023-01-25T05:43:45.409+0000 7f6cbca60700 -1 osd.5 43 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:45.523 INFO:tasks.ceph.osd.4.smithi203.stderr:2023-01-25T05:43:45.522+0000 7f984f629700 -1 osd.4 44 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:45.523 INFO:tasks.ceph.osd.2.smithi203.stderr:2023-01-25T05:43:45.522+0000 7f10cbec0700 -1 osd.2 44 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:45.524 INFO:tasks.ceph.osd.5.smithi203.stderr:2023-01-25T05:43:45.522+0000 7f6cade2d700 -1 osd.5 44 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:45.524 INFO:tasks.ceph.osd.0.smithi203.stderr:2023-01-25T05:43:45.522+0000 7f23bfa29700 -1 osd.0 44 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:45.525 INFO:tasks.ceph.osd.7.smithi203.stderr:2023-01-25T05:43:45.523+0000 7f3437e2c700 -1 osd.7 44 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:45.525 INFO:tasks.ceph.osd.3.smithi203.stderr:2023-01-25T05:43:45.523+0000 7f055e4e1700 -1 osd.3 44 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:45.744 INFO:tasks.workunit.client.0.smithi203.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2023-01-25T05:43:45.745 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health 2023-01-25T05:43:45.745 INFO:tasks.workunit.client.0.smithi203.stderr:+ grep '9 osds' 2023-01-25T05:43:46.241 INFO:tasks.workunit.client.0.smithi203.stdout:HEALTH_WARN noup,noout flag(s) set; 9 osds down; 4 hosts (8 osds) down; 2 racks (8 osds) down; 1 row (8 osds) down 2023-01-25T05:43:46.241 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph osd unset noup 2023-01-25T05:43:46.394 INFO:tasks.ceph.osd.1.smithi203.stderr:2023-01-25T05:43:46.392+0000 7f129dc54700 -1 osd.1 44 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:46.525 INFO:tasks.ceph.osd.5.smithi203.stderr:2023-01-25T05:43:46.524+0000 7f6cade2d700 -1 osd.5 45 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:46.526 INFO:tasks.ceph.osd.7.smithi203.stderr:2023-01-25T05:43:46.524+0000 7f3437e2c700 -1 osd.7 45 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:46.526 INFO:tasks.ceph.osd.0.smithi203.stderr:2023-01-25T05:43:46.525+0000 7f23bfa29700 -1 osd.0 45 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:46.527 INFO:tasks.ceph.osd.2.smithi203.stderr:2023-01-25T05:43:46.525+0000 7f10cbec0700 -1 osd.2 45 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:46.527 INFO:tasks.ceph.osd.4.smithi203.stderr:2023-01-25T05:43:46.525+0000 7f984f629700 -1 osd.4 45 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:46.528 INFO:tasks.ceph.osd.1.smithi203.stderr:2023-01-25T05:43:46.525+0000 7f128f822700 -1 osd.1 45 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:46.528 INFO:tasks.ceph.osd.3.smithi203.stderr:2023-01-25T05:43:46.525+0000 7f055e4e1700 -1 osd.3 45 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:47.490 INFO:tasks.ceph.osd.6.smithi203.stderr:2023-01-25T05:43:47.489+0000 7f0d3eda4700 -1 osd.6 45 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:47.531 INFO:tasks.ceph.osd.4.smithi203.stderr:2023-01-25T05:43:47.530+0000 7f9857639700 -1 osd.4 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:43:47.532 INFO:tasks.ceph.osd.3.smithi203.stderr:2023-01-25T05:43:47.530+0000 7f05664f1700 -1 osd.3 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:43:47.532 INFO:tasks.ceph.osd.7.smithi203.stderr:2023-01-25T05:43:47.531+0000 7f343fe3c700 -1 osd.7 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:43:47.533 INFO:tasks.ceph.osd.5.smithi203.stderr:2023-01-25T05:43:47.531+0000 7f6cb5e3d700 -1 osd.5 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:43:47.534 INFO:tasks.ceph.osd.2.smithi203.stderr:2023-01-25T05:43:47.531+0000 7f10d3ed0700 -1 osd.2 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:43:47.534 INFO:tasks.ceph.osd.0.smithi203.stderr:2023-01-25T05:43:47.532+0000 7f23c7a39700 -1 osd.0 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:43:47.535 INFO:tasks.ceph.osd.1.smithi203.stderr:2023-01-25T05:43:47.533+0000 7f1297832700 -1 osd.1 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:43:47.535 INFO:tasks.ceph.osd.6.smithi203.stderr:2023-01-25T05:43:47.533+0000 7f0d38181700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:43:48.014 INFO:tasks.ceph.osd.8.smithi203.stderr:2023-01-25T05:43:48.013+0000 7f75bc617700 -1 osd.8 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:43:48.562 INFO:tasks.workunit.client.0.smithi203.stderr:noup is unset 2023-01-25T05:43:48.574 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait_for_healthy 2023-01-25T05:43:48.574 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health 2023-01-25T05:43:48.575 INFO:tasks.workunit.client.0.smithi203.stderr:+ grep down 2023-01-25T05:43:49.087 INFO:tasks.workunit.client.0.smithi203.stderr:+ test_mark_two_osds_same_host_down_with_classes 2023-01-25T05:43:49.087 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph osd set noup 2023-01-25T05:43:50.556 INFO:tasks.workunit.client.0.smithi203.stderr:noup is set 2023-01-25T05:43:50.568 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph osd crush set-device-class ssd osd.0 osd.2 osd.4 osd.6 osd.8 2023-01-25T05:43:52.559 INFO:tasks.workunit.client.0.smithi203.stderr:osd.0 already set to class ssd. set-device-class item id 0 name 'osd.0' device_class 'ssd': no change. osd.2 already set to class ssd. set-device-class item id 2 name 'osd.2' device_class 'ssd': no change. osd.4 already set to class ssd. set-device-class item id 4 name 'osd.4' device_class 'ssd': no change. osd.6 already set to class ssd. set-device-class item id 6 name 'osd.6' device_class 'ssd': no change. osd.8 already set to class ssd. set-device-class item id 8 name 'osd.8' device_class 'ssd': no change. set osd(s) to class 'ssd' 2023-01-25T05:43:52.570 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph osd crush set-device-class hdd osd.1 osd.3 osd.5 osd.7 osd.9 2023-01-25T05:43:54.570 INFO:tasks.workunit.client.0.smithi203.stderr:osd.1 already set to class hdd. set-device-class item id 1 name 'osd.1' device_class 'hdd': no change. osd.3 already set to class hdd. set-device-class item id 3 name 'osd.3' device_class 'hdd': no change. osd.5 already set to class hdd. set-device-class item id 5 name 'osd.5' device_class 'hdd': no change. osd.7 already set to class hdd. set-device-class item id 7 name 'osd.7' device_class 'hdd': no change. osd.9 already set to class hdd. set-device-class item id 9 name 'osd.9' device_class 'hdd': no change. set osd(s) to class 'hdd' 2023-01-25T05:43:54.581 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph osd down osd.0 osd.1 2023-01-25T05:43:54.887 INFO:tasks.ceph.mon.a.smithi203.stderr:2023-01-25T05:43:54.887+0000 7f0213607700 -1 mon.a@0(leader).osd e53 definitely_dead 0 2023-01-25T05:43:55.701 INFO:tasks.ceph.mon.a.smithi203.stderr:2023-01-25T05:43:55.700+0000 7f0213607700 -1 mon.a@0(leader).osd e54 definitely_dead 0 2023-01-25T05:43:55.703 INFO:tasks.workunit.client.0.smithi203.stderr:osd.0 is already down. osd.1 is already down. 2023-01-25T05:43:55.713 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health detail 2023-01-25T05:43:56.266 INFO:tasks.workunit.client.0.smithi203.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2023-01-25T05:43:56.266 INFO:tasks.workunit.client.0.smithi203.stdout:[WRN] OSDMAP_FLAGS: noup,noout flag(s) set 2023-01-25T05:43:56.266 INFO:tasks.workunit.client.0.smithi203.stdout:[WRN] OSD_DOWN: 2 osds down 2023-01-25T05:43:56.267 INFO:tasks.workunit.client.0.smithi203.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2023-01-25T05:43:56.267 INFO:tasks.workunit.client.0.smithi203.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2023-01-25T05:43:56.267 INFO:tasks.workunit.client.0.smithi203.stdout:[WRN] OSD_HOST_DOWN: 1 host (2 osds) down 2023-01-25T05:43:56.267 INFO:tasks.workunit.client.0.smithi203.stdout: host host0 (root=root,row=row0,rack=rack0) (2 osds) is down 2023-01-25T05:43:56.279 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health 2023-01-25T05:43:56.279 INFO:tasks.workunit.client.0.smithi203.stderr:+ grep '1 host' 2023-01-25T05:43:56.779 INFO:tasks.workunit.client.0.smithi203.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2023-01-25T05:43:56.780 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health 2023-01-25T05:43:56.780 INFO:tasks.workunit.client.0.smithi203.stderr:+ grep '2 osds' 2023-01-25T05:43:57.290 INFO:tasks.workunit.client.0.smithi203.stdout:HEALTH_WARN noup,noout flag(s) set; 2 osds down; 1 host (2 osds) down 2023-01-25T05:43:57.291 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health detail 2023-01-25T05:43:57.291 INFO:tasks.workunit.client.0.smithi203.stderr:+ grep osd.0 2023-01-25T05:43:57.352 INFO:tasks.ceph.osd.0.smithi203.stderr:2023-01-25T05:43:57.351+0000 7f23cde5b700 -1 osd.0 54 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:57.595 INFO:tasks.ceph.osd.0.smithi203.stderr:2023-01-25T05:43:57.594+0000 7f23bfa29700 -1 osd.0 55 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:57.859 INFO:tasks.workunit.client.0.smithi203.stdout: osd.0 (root=root,row=row0,rack=rack0,host=host0) is down 2023-01-25T05:43:57.859 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health detail 2023-01-25T05:43:57.860 INFO:tasks.workunit.client.0.smithi203.stderr:+ grep osd.1 2023-01-25T05:43:58.430 INFO:tasks.workunit.client.0.smithi203.stdout: osd.1 (root=root,row=row0,rack=rack0,host=host0) is down 2023-01-25T05:43:58.430 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph osd unset noup 2023-01-25T05:43:58.606 INFO:tasks.ceph.osd.0.smithi203.stderr:2023-01-25T05:43:58.604+0000 7f23bfa29700 -1 osd.0 56 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:59.362 INFO:tasks.ceph.osd.1.smithi203.stderr:2023-01-25T05:43:59.361+0000 7f129e455700 -1 osd.1 56 osdmap NOUP flag is set, waiting for it to clear 2023-01-25T05:43:59.605 INFO:tasks.ceph.osd.1.smithi203.stderr:2023-01-25T05:43:59.604+0000 7f1297832700 -1 osd.1 57 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:43:59.606 INFO:tasks.ceph.osd.0.smithi203.stderr:2023-01-25T05:43:59.604+0000 7f23c7a39700 -1 osd.0 57 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-25T05:44:00.616 INFO:tasks.workunit.client.0.smithi203.stderr:noup is unset 2023-01-25T05:44:00.629 INFO:tasks.workunit.client.0.smithi203.stderr:+ wait_for_healthy 2023-01-25T05:44:00.629 INFO:tasks.workunit.client.0.smithi203.stderr:+ ceph health 2023-01-25T05:44:00.630 INFO:tasks.workunit.client.0.smithi203.stderr:+ grep down 2023-01-25T05:44:01.133 INFO:tasks.workunit.client.0.smithi203.stderr:+ exit 0 2023-01-25T05:44:01.133 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-01-25T05:44:01.133 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-01-25T05:44:01.200 INFO:tasks.workunit:Stopping ['rados/test_health_warnings.sh'] on client.0... 2023-01-25T05:44:01.201 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-01-25T05:44:01.511 DEBUG:teuthology.parallel:result is None 2023-01-25T05:44:01.512 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-01-25T05:44:01.539 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-01-25T05:44:01.540 DEBUG:teuthology.orchestra.run.smithi203:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-01-25T05:44:01.598 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-01-25T05:44:01.598 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-01-25T05:44:01.608 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-01-25T05:44:01.609 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-25T05:44:01.955 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:44:01.955 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":59,"fsid":"5b00363d-2783-4934-a70f-2b1720d57fe5","created":"2023-01-25T05:42:20.347257+0000","modified":"2023-01-25T05:44:01.606698+0000","last_up_change":"2023-01-25T05:44:00.605730+0000","last_in_change":"2023-01-25T05:42:23.705123+0000","flags":"noout,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5800064,"flags_set":["noout","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":10,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":10,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-01-25T05:42:32.990588+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"f87dc56d-86e7-496d-8e8f-b0d7dbca30a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":57,"up_from":58,"up_thru":34,"down_at":54,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":110629},{"type":"v1","addr":"172.21.15.203:6809","nonce":110629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6874","nonce":3110629},{"type":"v1","addr":"172.21.15.203:6875","nonce":3110629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":110629},{"type":"v1","addr":"172.21.15.203:6815","nonce":110629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":110629},{"type":"v1","addr":"172.21.15.203:6813","nonce":110629}]},"public_addr":"172.21.15.203:6809/110629","cluster_addr":"172.21.15.203:6875/3110629","heartbeat_back_addr":"172.21.15.203:6815/110629","heartbeat_front_addr":"172.21.15.203:6813/110629","state":["exists","up"]},{"osd":1,"uuid":"f35f7666-378a-4cc5-bd86-fd0f0a2e1b31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":57,"up_from":58,"up_thru":58,"down_at":54,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":110637},{"type":"v1","addr":"172.21.15.203:6825","nonce":110637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3110637},{"type":"v1","addr":"172.21.15.203:6803","nonce":3110637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":110637},{"type":"v1","addr":"172.21.15.203:6831","nonce":110637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":110637},{"type":"v1","addr":"172.21.15.203:6829","nonce":110637}]},"public_addr":"172.21.15.203:6825/110637","cluster_addr":"172.21.15.203:6803/3110637","heartbeat_back_addr":"172.21.15.203:6831/110637","heartbeat_front_addr":"172.21.15.203:6829/110637","state":["exists","up"]},{"osd":2,"uuid":"e5ea1037-85eb-494e-b195-161cd77cb6fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":46,"up_from":47,"up_thru":0,"down_at":42,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":110630},{"type":"v1","addr":"172.21.15.203:6817","nonce":110630}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":1110630},{"type":"v1","addr":"172.21.15.203:6835","nonce":1110630}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":110630},{"type":"v1","addr":"172.21.15.203:6823","nonce":110630}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":110630},{"type":"v1","addr":"172.21.15.203:6821","nonce":110630}]},"public_addr":"172.21.15.203:6817/110630","cluster_addr":"172.21.15.203:6835/1110630","heartbeat_back_addr":"172.21.15.203:6823/110630","heartbeat_front_addr":"172.21.15.203:6821/110630","state":["exists","up"]},{"osd":3,"uuid":"3451afca-3f67-4a33-88bc-9c4a3fa71b37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":46,"up_from":47,"up_thru":58,"down_at":42,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":110642},{"type":"v1","addr":"172.21.15.203:6801","nonce":110642}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6849","nonce":1110642},{"type":"v1","addr":"172.21.15.203:6851","nonce":1110642}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":110642},{"type":"v1","addr":"172.21.15.203:6807","nonce":110642}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":110642},{"type":"v1","addr":"172.21.15.203:6805","nonce":110642}]},"public_addr":"172.21.15.203:6801/110642","cluster_addr":"172.21.15.203:6851/1110642","heartbeat_back_addr":"172.21.15.203:6807/110642","heartbeat_front_addr":"172.21.15.203:6805/110642","state":["exists","up"]},{"osd":4,"uuid":"53249305-7472-4032-88c6-b6f2eab2f939","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":46,"up_from":47,"up_thru":58,"down_at":42,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6845","nonce":110696},{"type":"v1","addr":"172.21.15.203:6847","nonce":110696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6858","nonce":1110696},{"type":"v1","addr":"172.21.15.203:6860","nonce":1110696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6854","nonce":110696},{"type":"v1","addr":"172.21.15.203:6855","nonce":110696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6852","nonce":110696},{"type":"v1","addr":"172.21.15.203:6853","nonce":110696}]},"public_addr":"172.21.15.203:6847/110696","cluster_addr":"172.21.15.203:6860/1110696","heartbeat_back_addr":"172.21.15.203:6855/110696","heartbeat_front_addr":"172.21.15.203:6853/110696","state":["exists","up"]},{"osd":5,"uuid":"3c6cd644-1d97-4a74-815a-cbffa9685718","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":46,"up_from":47,"up_thru":47,"down_at":42,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":110644},{"type":"v1","addr":"172.21.15.203:6833","nonce":110644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6864","nonce":1110644},{"type":"v1","addr":"172.21.15.203:6866","nonce":1110644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":110644},{"type":"v1","addr":"172.21.15.203:6840","nonce":110644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":110644},{"type":"v1","addr":"172.21.15.203:6837","nonce":110644}]},"public_addr":"172.21.15.203:6833/110644","cluster_addr":"172.21.15.203:6866/1110644","heartbeat_back_addr":"172.21.15.203:6840/110644","heartbeat_front_addr":"172.21.15.203:6837/110644","state":["exists","up"]},{"osd":6,"uuid":"1426af76-6197-499f-8aa3-9202c91b6fc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":46,"up_from":47,"up_thru":54,"down_at":42,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6839","nonce":110685},{"type":"v1","addr":"172.21.15.203:6841","nonce":110685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":1110685},{"type":"v1","addr":"172.21.15.203:6811","nonce":1110685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6848","nonce":110685},{"type":"v1","addr":"172.21.15.203:6850","nonce":110685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6844","nonce":110685},{"type":"v1","addr":"172.21.15.203:6846","nonce":110685}]},"public_addr":"172.21.15.203:6841/110685","cluster_addr":"172.21.15.203:6811/1110685","heartbeat_back_addr":"172.21.15.203:6850/110685","heartbeat_front_addr":"172.21.15.203:6846/110685","state":["exists","up"]},{"osd":7,"uuid":"a13fb8e1-a46d-4398-956d-ce4d7b04da48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":46,"up_from":47,"up_thru":47,"down_at":42,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6859","nonce":110783},{"type":"v1","addr":"172.21.15.203:6861","nonce":110783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6882","nonce":1110783},{"type":"v1","addr":"172.21.15.203:6883","nonce":1110783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6870","nonce":110783},{"type":"v1","addr":"172.21.15.203:6871","nonce":110783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6868","nonce":110783},{"type":"v1","addr":"172.21.15.203:6869","nonce":110783}]},"public_addr":"172.21.15.203:6861/110783","cluster_addr":"172.21.15.203:6883/1110783","heartbeat_back_addr":"172.21.15.203:6871/110783","heartbeat_front_addr":"172.21.15.203:6869/110783","state":["exists","up"]},{"osd":8,"uuid":"cc46f5f4-d476-4ff6-b7f9-537263cb24cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":46,"up_from":47,"up_thru":17,"down_at":42,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6856","nonce":110781},{"type":"v1","addr":"172.21.15.203:6857","nonce":110781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6842","nonce":2110781},{"type":"v1","addr":"172.21.15.203:6843","nonce":2110781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6865","nonce":110781},{"type":"v1","addr":"172.21.15.203:6867","nonce":110781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6862","nonce":110781},{"type":"v1","addr":"172.21.15.203:6863","nonce":110781}]},"public_addr":"172.21.15.203:6857/110781","cluster_addr":"172.21.15.203:6843/2110781","heartbeat_back_addr":"172.21.15.203:6867/110781","heartbeat_front_addr":"172.21.15.203:6863/110781","state":["exists","up"]},{"osd":9,"uuid":"2ac16a61-4dcd-43be-a686-f9959a8901ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":38,"up_from":39,"up_thru":58,"down_at":34,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6872","nonce":110792},{"type":"v1","addr":"172.21.15.203:6873","nonce":110792}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1110792},{"type":"v1","addr":"172.21.15.203:6827","nonce":1110792}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6878","nonce":110792},{"type":"v1","addr":"172.21.15.203:6879","nonce":110792}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6876","nonce":110792},{"type":"v1","addr":"172.21.15.203:6877","nonce":110792}]},"public_addr":"172.21.15.203:6873/110792","cluster_addr":"172.21.15.203:6827/1110792","heartbeat_back_addr":"172.21.15.203:6879/110792","heartbeat_front_addr":"172.21.15.203:6877/110792","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"2023-01-25T05:43:55.560430+0000","laggy_probability":0.65700000524520874,"laggy_interval":1,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:27.924170+0000","dead_epoch":54},{"osd":1,"down_stamp":"2023-01-25T05:43:55.560430+0000","laggy_probability":0.65700000524520874,"laggy_interval":1,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.162198+0000","dead_epoch":55},{"osd":2,"down_stamp":"2023-01-25T05:43:43.512204+0000","laggy_probability":0.30000001192092896,"laggy_interval":1,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.104251+0000","dead_epoch":43},{"osd":3,"down_stamp":"2023-01-25T05:43:43.512204+0000","laggy_probability":0.30000001192092896,"laggy_interval":1,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:27.901694+0000","dead_epoch":42},{"osd":4,"down_stamp":"2023-01-25T05:43:43.512204+0000","laggy_probability":0.30000001192092896,"laggy_interval":1,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.209516+0000","dead_epoch":42},{"osd":5,"down_stamp":"2023-01-25T05:43:43.512204+0000","laggy_probability":0.30000001192092896,"laggy_interval":1,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.128175+0000","dead_epoch":43},{"osd":6,"down_stamp":"2023-01-25T05:43:43.512204+0000","laggy_probability":0.30000001192092896,"laggy_interval":1,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.144209+0000","dead_epoch":45},{"osd":7,"down_stamp":"2023-01-25T05:43:43.512204+0000","laggy_probability":0.30000001192092896,"laggy_interval":1,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.228683+0000","dead_epoch":42},{"osd":8,"down_stamp":"2023-01-25T05:43:43.512204+0000","laggy_probability":0.50999999046325684,"laggy_interval":2,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.195999+0000","dead_epoch":45},{"osd":9,"down_stamp":"2023-01-25T05:43:34.475739+0000","laggy_probability":0.30000001192092896,"laggy_interval":1,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-25T05:42:28.258473+0000","dead_epoch":35}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-25T05:44:01.970 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-01-25T05:44:01.970 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-25T05:44:02.283 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:44:02.286 INFO:teuthology.orchestra.run.smithi203.stderr:dumped all 2023-01-25T05:44:02.297 INFO:teuthology.orchestra.run.smithi203.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2023-01-25T05:44:02.107123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":58840,"kb_used_data":6216,"kb_used_omap":0,"kb_used_meta":52544,"kb_avail":1048517160,"statfs":{"total":1073741824000,"available":1073681571840,"internally_reserved":0,"allocated":6365184,"data_stored":3020570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53805056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":45,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-01-25T05:43:57.845120+0000","last_change":"2023-01-25T05:43:49.550719+0000","last_active":"2023-01-25T05:43:57.845120+0000","last_peered":"2023-01-25T05:43:57.845120+0000","last_clean":"2023-01-25T05:43:57.845120+0000","last_became_active":"2023-01-25T05:43:49.550449+0000","last_became_peered":"2023-01-25T05:43:49.550449+0000","last_unstale":"2023-01-25T05:43:57.845120+0000","last_undegraded":"2023-01-25T05:43:57.845120+0000","last_fullsized":"2023-01-25T05:43:57.845120+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T06:44:14.717000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-01-25T05:43:57.599466+0000","last_change":"2023-01-25T05:43:49.546867+0000","last_active":"2023-01-25T05:43:57.599466+0000","last_peered":"2023-01-25T05:43:57.599466+0000","last_clean":"2023-01-25T05:43:57.599466+0000","last_became_active":"2023-01-25T05:43:49.546602+0000","last_became_peered":"2023-01-25T05:43:49.546602+0000","last_unstale":"2023-01-25T05:43:57.599466+0000","last_undegraded":"2023-01-25T05:43:57.599466+0000","last_fullsized":"2023-01-25T05:43:57.599466+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T15:50:01.461764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-01-25T05:43:59.516529+0000","last_change":"2023-01-25T05:43:49.548900+0000","last_active":"2023-01-25T05:43:59.516529+0000","last_peered":"2023-01-25T05:43:59.516529+0000","last_clean":"2023-01-25T05:43:59.516529+0000","last_became_active":"2023-01-25T05:43:49.548607+0000","last_became_peered":"2023-01-25T05:43:49.548607+0000","last_unstale":"2023-01-25T05:43:59.516529+0000","last_undegraded":"2023-01-25T05:43:59.516529+0000","last_fullsized":"2023-01-25T05:43:59.516529+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T13:10:06.551804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-01-25T05:43:59.240067+0000","last_change":"2023-01-25T05:43:49.546727+0000","last_active":"2023-01-25T05:43:59.240067+0000","last_peered":"2023-01-25T05:43:59.240067+0000","last_clean":"2023-01-25T05:43:59.240067+0000","last_became_active":"2023-01-25T05:43:49.546464+0000","last_became_peered":"2023-01-25T05:43:49.546464+0000","last_unstale":"2023-01-25T05:43:59.240067+0000","last_undegraded":"2023-01-25T05:43:59.240067+0000","last_fullsized":"2023-01-25T05:43:59.240067+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T10:57:55.038293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":66,"reported_epoch":56,"state":"active+undersized","last_fresh":"2023-01-25T05:43:58.608440+0000","last_change":"2023-01-25T05:43:58.608406+0000","last_active":"2023-01-25T05:43:58.608440+0000","last_peered":"2023-01-25T05:43:58.608440+0000","last_clean":"2023-01-25T05:43:55.402337+0000","last_became_active":"2023-01-25T05:43:58.608406+0000","last_became_peered":"2023-01-25T05:43:58.608406+0000","last_unstale":"2023-01-25T05:43:58.608440+0000","last_undegraded":"2023-01-25T05:43:58.608440+0000","last_fullsized":"2023-01-25T05:43:58.608057+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T08:55:52.504843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":70,"reported_epoch":56,"state":"active+undersized","last_fresh":"2023-01-25T05:43:58.607256+0000","last_change":"2023-01-25T05:43:58.607191+0000","last_active":"2023-01-25T05:43:58.607256+0000","last_peered":"2023-01-25T05:43:58.607256+0000","last_clean":"2023-01-25T05:43:55.402545+0000","last_became_active":"2023-01-25T05:43:58.607191+0000","last_became_peered":"2023-01-25T05:43:58.607191+0000","last_unstale":"2023-01-25T05:43:58.607256+0000","last_undegraded":"2023-01-25T05:43:58.607256+0000","last_fullsized":"2023-01-25T05:43:58.605244+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T09:57:00.276783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":56,"state":"active+undersized","last_fresh":"2023-01-25T05:43:59.516475+0000","last_change":"2023-01-25T05:43:59.516475+0000","last_active":"2023-01-25T05:43:59.516475+0000","last_peered":"2023-01-25T05:43:59.516475+0000","last_clean":"2023-01-25T05:43:43.709021+0000","last_became_active":"2023-01-25T05:43:57.599721+0000","last_became_peered":"2023-01-25T05:43:57.599721+0000","last_unstale":"2023-01-25T05:43:59.516475+0000","last_undegraded":"2023-01-25T05:43:59.516475+0000","last_fullsized":"2023-01-25T05:43:57.599150+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T14:03:23.649310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":86,"reported_epoch":56,"state":"active+undersized","last_fresh":"2023-01-25T05:43:58.608362+0000","last_change":"2023-01-25T05:43:58.608316+0000","last_active":"2023-01-25T05:43:58.608362+0000","last_peered":"2023-01-25T05:43:58.608362+0000","last_clean":"2023-01-25T05:43:55.048820+0000","last_became_active":"2023-01-25T05:43:58.608316+0000","last_became_peered":"2023-01-25T05:43:58.608316+0000","last_unstale":"2023-01-25T05:43:58.608362+0000","last_undegraded":"2023-01-25T05:43:58.608362+0000","last_fullsized":"2023-01-25T05:43:58.607892+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T09:10:36.419019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":39,"seq":167503724564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5880,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851720,"statfs":{"total":107374182400,"available":107368161280,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":2,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.62},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.42199999999999999}]},{"osd":3,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.57299999999999995}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.502},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.67800000000000005}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.64000000000000001}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.78000000000000003}]},{"osd":7,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.73299999999999998}]},{"osd":8,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.754}]}]},{"osd":8,"up_from":47,"seq":201863462932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":5804,"kb_used_data":612,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":104851796,"statfs":{"total":107374182400,"available":107368239104,"internally_reserved":0,"allocated":626688,"data_stored":295889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":2,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.313},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.45100000000000001}]},{"osd":3,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.5},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.46000000000000002}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.43099999999999999}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.40600000000000003}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.51400000000000001}]},{"osd":7,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.52600000000000002}]},{"osd":9,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.56000000000000005}]}]},{"osd":7,"up_from":47,"seq":201863462933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5880,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851720,"statfs":{"total":107374182400,"available":107368161280,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":2,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.30599999999999999}]},{"osd":3,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.38400000000000001}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.42399999999999999}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.44}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.58499999999999996}]},{"osd":8,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.505},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.55800000000000005}]},{"osd":9,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.47399999999999998}]}]},{"osd":6,"up_from":47,"seq":201863462932,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5880,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851720,"statfs":{"total":107374182400,"available":107368161280,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":2,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":7.7309999999999999,"5min":7.7309999999999999,"15min":7.7309999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":2.5139999999999998,"5min":2.5139999999999998,"15min":2.5139999999999998},"last":0.56599999999999995}]},{"osd":3,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":2.7549999999999999,"5min":2.7549999999999999,"15min":2.7549999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":7.7729999999999997,"5min":7.7729999999999997,"15min":7.7729999999999997},"last":0.63300000000000001}]},{"osd":4,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":2.5350000000000001,"5min":2.5350000000000001,"15min":2.5350000000000001},"last":0.496},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.7799999999999998,"5min":2.7799999999999998,"15min":2.7799999999999998},"last":0.621}]},{"osd":5,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":7.7990000000000004,"5min":7.7990000000000004,"15min":7.7990000000000004},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.4569999999999999,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":0.64900000000000002}]},{"osd":7,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":7.8250000000000002,"5min":7.8250000000000002,"15min":7.8250000000000002},"last":0.67900000000000005}]},{"osd":8,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":2.556,"5min":2.556,"15min":2.556},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":2.73,"5min":2.73,"15min":2.73},"last":0.70499999999999996}]},{"osd":9,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":7.8499999999999996,"5min":7.8499999999999996,"15min":7.8499999999999996},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.5840000000000001,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":0.72699999999999998}]}]},{"osd":1,"up_from":0,"seq":21,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5880,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851720,"statfs":{"total":107374182400,"available":107368161280,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.512}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.673}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7370000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.802}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3799999999999999}]}]},{"osd":0,"up_from":0,"seq":20,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5944,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":104851656,"statfs":{"total":107374182400,"available":107368095744,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.216}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]}]},{"osd":2,"up_from":47,"seq":201863462932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":5868,"kb_used_data":612,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851732,"statfs":{"total":107374182400,"available":107368173568,"internally_reserved":0,"allocated":626688,"data_stored":295889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":3,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.52600000000000002}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.51900000000000002}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.55300000000000005}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.393,"5min":1.393,"15min":1.393},"last":0.50900000000000001}]},{"osd":7,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.53500000000000003}]},{"osd":8,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.42299999999999999}]},{"osd":9,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.5},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":0.71799999999999997}]}]},{"osd":3,"up_from":47,"seq":201863462932,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5944,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":104851656,"statfs":{"total":107374182400,"available":107368095744,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":2,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.9319999999999999,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.1960000000000002,"5min":2.1960000000000002,"15min":2.1960000000000002},"last":0.46600000000000003}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.8979999999999999,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":0.55200000000000005}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.492,"5min":3.492,"15min":3.492},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":3.3340000000000001,"5min":3.3340000000000001,"15min":3.3340000000000001},"last":0.38100000000000001}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.226,"5min":2.226,"15min":2.226},"last":0.42499999999999999}]},{"osd":7,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.50700000000000001}]},{"osd":8,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":2.359,"5min":2.359,"15min":2.359},"last":0.443},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.53200000000000003}]},{"osd":9,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":2.839,"5min":2.839,"15min":2.839},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":0.47299999999999998}]}]},{"osd":4,"up_from":47,"seq":201863462932,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5880,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851720,"statfs":{"total":107374182400,"available":107368161280,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":2,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.3119999999999998,"5min":3.3119999999999998,"15min":3.3119999999999998},"last":0.40400000000000003}]},{"osd":3,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.82699999999999996}]},{"osd":5,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":5.835,"5min":5.835,"15min":5.835},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.66000000000000003}]},{"osd":6,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":2.613,"5min":2.613,"15min":2.613},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.4470000000000001,"5min":3.4470000000000001,"15min":3.4470000000000001},"last":0.64800000000000002}]},{"osd":7,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.649,"5min":1.649,"15min":1.649},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":0.68899999999999995}]},{"osd":8,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":3.117,"5min":3.117,"15min":3.117},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"last":0.85699999999999998}]},{"osd":9,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":3.1480000000000001,"5min":3.1480000000000001,"15min":3.1480000000000001},"last":0.52800000000000002}]}]},{"osd":5,"up_from":47,"seq":201863462933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5880,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851720,"statfs":{"total":107374182400,"available":107368161280,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":2,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.36099999999999999}]},{"osd":3,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.40999999999999998}]},{"osd":4,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.55600000000000005}]},{"osd":6,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.42199999999999999}]},{"osd":7,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.44800000000000001}]},{"osd":8,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.495},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.58799999999999997}]},{"osd":9,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.53900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-25T05:44:02.300 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-25T05:44:02.613 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:44:02.616 INFO:teuthology.orchestra.run.smithi203.stderr:dumped all 2023-01-25T05:44:02.628 INFO:teuthology.orchestra.run.smithi203.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2023-01-25T05:44:02.107123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":58840,"kb_used_data":6216,"kb_used_omap":0,"kb_used_meta":52544,"kb_avail":1048517160,"statfs":{"total":1073741824000,"available":1073681571840,"internally_reserved":0,"allocated":6365184,"data_stored":3020570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53805056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":45,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-01-25T05:43:57.845120+0000","last_change":"2023-01-25T05:43:49.550719+0000","last_active":"2023-01-25T05:43:57.845120+0000","last_peered":"2023-01-25T05:43:57.845120+0000","last_clean":"2023-01-25T05:43:57.845120+0000","last_became_active":"2023-01-25T05:43:49.550449+0000","last_became_peered":"2023-01-25T05:43:49.550449+0000","last_unstale":"2023-01-25T05:43:57.845120+0000","last_undegraded":"2023-01-25T05:43:57.845120+0000","last_fullsized":"2023-01-25T05:43:57.845120+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T06:44:14.717000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-01-25T05:43:57.599466+0000","last_change":"2023-01-25T05:43:49.546867+0000","last_active":"2023-01-25T05:43:57.599466+0000","last_peered":"2023-01-25T05:43:57.599466+0000","last_clean":"2023-01-25T05:43:57.599466+0000","last_became_active":"2023-01-25T05:43:49.546602+0000","last_became_peered":"2023-01-25T05:43:49.546602+0000","last_unstale":"2023-01-25T05:43:57.599466+0000","last_undegraded":"2023-01-25T05:43:57.599466+0000","last_fullsized":"2023-01-25T05:43:57.599466+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T15:50:01.461764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-01-25T05:43:59.516529+0000","last_change":"2023-01-25T05:43:49.548900+0000","last_active":"2023-01-25T05:43:59.516529+0000","last_peered":"2023-01-25T05:43:59.516529+0000","last_clean":"2023-01-25T05:43:59.516529+0000","last_became_active":"2023-01-25T05:43:49.548607+0000","last_became_peered":"2023-01-25T05:43:49.548607+0000","last_unstale":"2023-01-25T05:43:59.516529+0000","last_undegraded":"2023-01-25T05:43:59.516529+0000","last_fullsized":"2023-01-25T05:43:59.516529+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T13:10:06.551804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-01-25T05:43:59.240067+0000","last_change":"2023-01-25T05:43:49.546727+0000","last_active":"2023-01-25T05:43:59.240067+0000","last_peered":"2023-01-25T05:43:59.240067+0000","last_clean":"2023-01-25T05:43:59.240067+0000","last_became_active":"2023-01-25T05:43:49.546464+0000","last_became_peered":"2023-01-25T05:43:49.546464+0000","last_unstale":"2023-01-25T05:43:59.240067+0000","last_undegraded":"2023-01-25T05:43:59.240067+0000","last_fullsized":"2023-01-25T05:43:59.240067+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T10:57:55.038293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":66,"reported_epoch":56,"state":"active+undersized","last_fresh":"2023-01-25T05:43:58.608440+0000","last_change":"2023-01-25T05:43:58.608406+0000","last_active":"2023-01-25T05:43:58.608440+0000","last_peered":"2023-01-25T05:43:58.608440+0000","last_clean":"2023-01-25T05:43:55.402337+0000","last_became_active":"2023-01-25T05:43:58.608406+0000","last_became_peered":"2023-01-25T05:43:58.608406+0000","last_unstale":"2023-01-25T05:43:58.608440+0000","last_undegraded":"2023-01-25T05:43:58.608440+0000","last_fullsized":"2023-01-25T05:43:58.608057+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T08:55:52.504843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":70,"reported_epoch":56,"state":"active+undersized","last_fresh":"2023-01-25T05:43:58.607256+0000","last_change":"2023-01-25T05:43:58.607191+0000","last_active":"2023-01-25T05:43:58.607256+0000","last_peered":"2023-01-25T05:43:58.607256+0000","last_clean":"2023-01-25T05:43:55.402545+0000","last_became_active":"2023-01-25T05:43:58.607191+0000","last_became_peered":"2023-01-25T05:43:58.607191+0000","last_unstale":"2023-01-25T05:43:58.607256+0000","last_undegraded":"2023-01-25T05:43:58.607256+0000","last_fullsized":"2023-01-25T05:43:58.605244+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T09:57:00.276783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":56,"state":"active+undersized","last_fresh":"2023-01-25T05:43:59.516475+0000","last_change":"2023-01-25T05:43:59.516475+0000","last_active":"2023-01-25T05:43:59.516475+0000","last_peered":"2023-01-25T05:43:59.516475+0000","last_clean":"2023-01-25T05:43:43.709021+0000","last_became_active":"2023-01-25T05:43:57.599721+0000","last_became_peered":"2023-01-25T05:43:57.599721+0000","last_unstale":"2023-01-25T05:43:59.516475+0000","last_undegraded":"2023-01-25T05:43:59.516475+0000","last_fullsized":"2023-01-25T05:43:57.599150+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T14:03:23.649310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":86,"reported_epoch":56,"state":"active+undersized","last_fresh":"2023-01-25T05:43:58.608362+0000","last_change":"2023-01-25T05:43:58.608316+0000","last_active":"2023-01-25T05:43:58.608362+0000","last_peered":"2023-01-25T05:43:58.608362+0000","last_clean":"2023-01-25T05:43:55.048820+0000","last_became_active":"2023-01-25T05:43:58.608316+0000","last_became_peered":"2023-01-25T05:43:58.608316+0000","last_unstale":"2023-01-25T05:43:58.608362+0000","last_undegraded":"2023-01-25T05:43:58.608362+0000","last_fullsized":"2023-01-25T05:43:58.607892+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T09:10:36.419019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":39,"seq":167503724564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5880,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851720,"statfs":{"total":107374182400,"available":107368161280,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":2,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.62},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.42199999999999999}]},{"osd":3,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.57299999999999995}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.502},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.67800000000000005}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.64000000000000001}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.78000000000000003}]},{"osd":7,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.73299999999999998}]},{"osd":8,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.754}]}]},{"osd":8,"up_from":47,"seq":201863462932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":5804,"kb_used_data":612,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":104851796,"statfs":{"total":107374182400,"available":107368239104,"internally_reserved":0,"allocated":626688,"data_stored":295889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":2,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.313},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.45100000000000001}]},{"osd":3,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.5},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.46000000000000002}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.43099999999999999}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.40600000000000003}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.51400000000000001}]},{"osd":7,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.52600000000000002}]},{"osd":9,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.56000000000000005}]}]},{"osd":7,"up_from":47,"seq":201863462933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5880,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851720,"statfs":{"total":107374182400,"available":107368161280,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":2,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.30599999999999999}]},{"osd":3,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.38400000000000001}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.42399999999999999}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.44}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.58499999999999996}]},{"osd":8,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.505},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.55800000000000005}]},{"osd":9,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.47399999999999998}]}]},{"osd":6,"up_from":47,"seq":201863462932,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5880,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851720,"statfs":{"total":107374182400,"available":107368161280,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":2,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":7.7309999999999999,"5min":7.7309999999999999,"15min":7.7309999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":2.5139999999999998,"5min":2.5139999999999998,"15min":2.5139999999999998},"last":0.56599999999999995}]},{"osd":3,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":2.7549999999999999,"5min":2.7549999999999999,"15min":2.7549999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":7.7729999999999997,"5min":7.7729999999999997,"15min":7.7729999999999997},"last":0.63300000000000001}]},{"osd":4,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":2.5350000000000001,"5min":2.5350000000000001,"15min":2.5350000000000001},"last":0.496},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.7799999999999998,"5min":2.7799999999999998,"15min":2.7799999999999998},"last":0.621}]},{"osd":5,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":7.7990000000000004,"5min":7.7990000000000004,"15min":7.7990000000000004},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.4569999999999999,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":0.64900000000000002}]},{"osd":7,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":7.8250000000000002,"5min":7.8250000000000002,"15min":7.8250000000000002},"last":0.67900000000000005}]},{"osd":8,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":2.556,"5min":2.556,"15min":2.556},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":2.73,"5min":2.73,"15min":2.73},"last":0.70499999999999996}]},{"osd":9,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":7.8499999999999996,"5min":7.8499999999999996,"15min":7.8499999999999996},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.5840000000000001,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":0.72699999999999998}]}]},{"osd":1,"up_from":0,"seq":21,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5880,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851720,"statfs":{"total":107374182400,"available":107368161280,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.512}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.673}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7370000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.802}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3799999999999999}]}]},{"osd":0,"up_from":0,"seq":20,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5944,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":104851656,"statfs":{"total":107374182400,"available":107368095744,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.216}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]}]},{"osd":2,"up_from":47,"seq":201863462932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":5868,"kb_used_data":612,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851732,"statfs":{"total":107374182400,"available":107368173568,"internally_reserved":0,"allocated":626688,"data_stored":295889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":3,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.52600000000000002}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.51900000000000002}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.55300000000000005}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.393,"5min":1.393,"15min":1.393},"last":0.50900000000000001}]},{"osd":7,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.53500000000000003}]},{"osd":8,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.42299999999999999}]},{"osd":9,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.5},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":0.71799999999999997}]}]},{"osd":3,"up_from":47,"seq":201863462932,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5944,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":104851656,"statfs":{"total":107374182400,"available":107368095744,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":2,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.9319999999999999,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.1960000000000002,"5min":2.1960000000000002,"15min":2.1960000000000002},"last":0.46600000000000003}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.8979999999999999,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":0.55200000000000005}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.492,"5min":3.492,"15min":3.492},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":3.3340000000000001,"5min":3.3340000000000001,"15min":3.3340000000000001},"last":0.38100000000000001}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.226,"5min":2.226,"15min":2.226},"last":0.42499999999999999}]},{"osd":7,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.50700000000000001}]},{"osd":8,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":2.359,"5min":2.359,"15min":2.359},"last":0.443},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.53200000000000003}]},{"osd":9,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":2.839,"5min":2.839,"15min":2.839},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":0.47299999999999998}]}]},{"osd":4,"up_from":47,"seq":201863462932,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5880,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851720,"statfs":{"total":107374182400,"available":107368161280,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":2,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.3119999999999998,"5min":3.3119999999999998,"15min":3.3119999999999998},"last":0.40400000000000003}]},{"osd":3,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.82699999999999996}]},{"osd":5,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":5.835,"5min":5.835,"15min":5.835},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.66000000000000003}]},{"osd":6,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":2.613,"5min":2.613,"15min":2.613},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.4470000000000001,"5min":3.4470000000000001,"15min":3.4470000000000001},"last":0.64800000000000002}]},{"osd":7,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.649,"5min":1.649,"15min":1.649},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":0.68899999999999995}]},{"osd":8,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":3.117,"5min":3.117,"15min":3.117},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"last":0.85699999999999998}]},{"osd":9,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":3.1480000000000001,"5min":3.1480000000000001,"15min":3.1480000000000001},"last":0.52800000000000002}]}]},{"osd":5,"up_from":47,"seq":201863462933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5880,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851720,"statfs":{"total":107374182400,"available":107368161280,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":2,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.36099999999999999}]},{"osd":3,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.40999999999999998}]},{"osd":4,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.55600000000000005}]},{"osd":6,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.42199999999999999}]},{"osd":7,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.44800000000000001}]},{"osd":8,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.495},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.58799999999999997}]},{"osd":9,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.53900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-25T05:44:02.631 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-25T05:44:02.943 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:44:02.945 INFO:teuthology.orchestra.run.smithi203.stderr:dumped all 2023-01-25T05:44:02.956 INFO:teuthology.orchestra.run.smithi203.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2023-01-25T05:44:02.107123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":58840,"kb_used_data":6216,"kb_used_omap":0,"kb_used_meta":52544,"kb_avail":1048517160,"statfs":{"total":1073741824000,"available":1073681571840,"internally_reserved":0,"allocated":6365184,"data_stored":3020570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53805056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":45,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-01-25T05:43:57.845120+0000","last_change":"2023-01-25T05:43:49.550719+0000","last_active":"2023-01-25T05:43:57.845120+0000","last_peered":"2023-01-25T05:43:57.845120+0000","last_clean":"2023-01-25T05:43:57.845120+0000","last_became_active":"2023-01-25T05:43:49.550449+0000","last_became_peered":"2023-01-25T05:43:49.550449+0000","last_unstale":"2023-01-25T05:43:57.845120+0000","last_undegraded":"2023-01-25T05:43:57.845120+0000","last_fullsized":"2023-01-25T05:43:57.845120+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T06:44:14.717000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-01-25T05:43:57.599466+0000","last_change":"2023-01-25T05:43:49.546867+0000","last_active":"2023-01-25T05:43:57.599466+0000","last_peered":"2023-01-25T05:43:57.599466+0000","last_clean":"2023-01-25T05:43:57.599466+0000","last_became_active":"2023-01-25T05:43:49.546602+0000","last_became_peered":"2023-01-25T05:43:49.546602+0000","last_unstale":"2023-01-25T05:43:57.599466+0000","last_undegraded":"2023-01-25T05:43:57.599466+0000","last_fullsized":"2023-01-25T05:43:57.599466+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T15:50:01.461764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-01-25T05:43:59.516529+0000","last_change":"2023-01-25T05:43:49.548900+0000","last_active":"2023-01-25T05:43:59.516529+0000","last_peered":"2023-01-25T05:43:59.516529+0000","last_clean":"2023-01-25T05:43:59.516529+0000","last_became_active":"2023-01-25T05:43:49.548607+0000","last_became_peered":"2023-01-25T05:43:49.548607+0000","last_unstale":"2023-01-25T05:43:59.516529+0000","last_undegraded":"2023-01-25T05:43:59.516529+0000","last_fullsized":"2023-01-25T05:43:59.516529+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T13:10:06.551804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-01-25T05:43:59.240067+0000","last_change":"2023-01-25T05:43:49.546727+0000","last_active":"2023-01-25T05:43:59.240067+0000","last_peered":"2023-01-25T05:43:59.240067+0000","last_clean":"2023-01-25T05:43:59.240067+0000","last_became_active":"2023-01-25T05:43:49.546464+0000","last_became_peered":"2023-01-25T05:43:49.546464+0000","last_unstale":"2023-01-25T05:43:59.240067+0000","last_undegraded":"2023-01-25T05:43:59.240067+0000","last_fullsized":"2023-01-25T05:43:59.240067+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T10:57:55.038293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":66,"reported_epoch":56,"state":"active+undersized","last_fresh":"2023-01-25T05:43:58.608440+0000","last_change":"2023-01-25T05:43:58.608406+0000","last_active":"2023-01-25T05:43:58.608440+0000","last_peered":"2023-01-25T05:43:58.608440+0000","last_clean":"2023-01-25T05:43:55.402337+0000","last_became_active":"2023-01-25T05:43:58.608406+0000","last_became_peered":"2023-01-25T05:43:58.608406+0000","last_unstale":"2023-01-25T05:43:58.608440+0000","last_undegraded":"2023-01-25T05:43:58.608440+0000","last_fullsized":"2023-01-25T05:43:58.608057+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T08:55:52.504843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":70,"reported_epoch":56,"state":"active+undersized","last_fresh":"2023-01-25T05:43:58.607256+0000","last_change":"2023-01-25T05:43:58.607191+0000","last_active":"2023-01-25T05:43:58.607256+0000","last_peered":"2023-01-25T05:43:58.607256+0000","last_clean":"2023-01-25T05:43:55.402545+0000","last_became_active":"2023-01-25T05:43:58.607191+0000","last_became_peered":"2023-01-25T05:43:58.607191+0000","last_unstale":"2023-01-25T05:43:58.607256+0000","last_undegraded":"2023-01-25T05:43:58.607256+0000","last_fullsized":"2023-01-25T05:43:58.605244+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T09:57:00.276783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":56,"state":"active+undersized","last_fresh":"2023-01-25T05:43:59.516475+0000","last_change":"2023-01-25T05:43:59.516475+0000","last_active":"2023-01-25T05:43:59.516475+0000","last_peered":"2023-01-25T05:43:59.516475+0000","last_clean":"2023-01-25T05:43:43.709021+0000","last_became_active":"2023-01-25T05:43:57.599721+0000","last_became_peered":"2023-01-25T05:43:57.599721+0000","last_unstale":"2023-01-25T05:43:59.516475+0000","last_undegraded":"2023-01-25T05:43:59.516475+0000","last_fullsized":"2023-01-25T05:43:57.599150+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T14:03:23.649310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":86,"reported_epoch":56,"state":"active+undersized","last_fresh":"2023-01-25T05:43:58.608362+0000","last_change":"2023-01-25T05:43:58.608316+0000","last_active":"2023-01-25T05:43:58.608362+0000","last_peered":"2023-01-25T05:43:58.608362+0000","last_clean":"2023-01-25T05:43:55.048820+0000","last_became_active":"2023-01-25T05:43:58.608316+0000","last_became_peered":"2023-01-25T05:43:58.608316+0000","last_unstale":"2023-01-25T05:43:58.608362+0000","last_undegraded":"2023-01-25T05:43:58.608362+0000","last_fullsized":"2023-01-25T05:43:58.607892+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T09:10:36.419019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":39,"seq":167503724564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5880,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851720,"statfs":{"total":107374182400,"available":107368161280,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":2,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.62},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.42199999999999999}]},{"osd":3,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.57299999999999995}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.502},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.67800000000000005}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.64000000000000001}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.78000000000000003}]},{"osd":7,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.73299999999999998}]},{"osd":8,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.754}]}]},{"osd":8,"up_from":47,"seq":201863462932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":5804,"kb_used_data":612,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":104851796,"statfs":{"total":107374182400,"available":107368239104,"internally_reserved":0,"allocated":626688,"data_stored":295889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":2,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.313},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.45100000000000001}]},{"osd":3,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.5},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.46000000000000002}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.43099999999999999}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.40600000000000003}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.51400000000000001}]},{"osd":7,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.52600000000000002}]},{"osd":9,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.56000000000000005}]}]},{"osd":7,"up_from":47,"seq":201863462933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5880,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851720,"statfs":{"total":107374182400,"available":107368161280,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":2,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.30599999999999999}]},{"osd":3,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.38400000000000001}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.42399999999999999}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.44}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.58499999999999996}]},{"osd":8,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.505},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.55800000000000005}]},{"osd":9,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.47399999999999998}]}]},{"osd":6,"up_from":47,"seq":201863462932,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5880,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851720,"statfs":{"total":107374182400,"available":107368161280,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":2,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":7.7309999999999999,"5min":7.7309999999999999,"15min":7.7309999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":2.5139999999999998,"5min":2.5139999999999998,"15min":2.5139999999999998},"last":0.56599999999999995}]},{"osd":3,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":2.7549999999999999,"5min":2.7549999999999999,"15min":2.7549999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":7.7729999999999997,"5min":7.7729999999999997,"15min":7.7729999999999997},"last":0.63300000000000001}]},{"osd":4,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":2.5350000000000001,"5min":2.5350000000000001,"15min":2.5350000000000001},"last":0.496},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.7799999999999998,"5min":2.7799999999999998,"15min":2.7799999999999998},"last":0.621}]},{"osd":5,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":7.7990000000000004,"5min":7.7990000000000004,"15min":7.7990000000000004},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.4569999999999999,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":0.64900000000000002}]},{"osd":7,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":7.8250000000000002,"5min":7.8250000000000002,"15min":7.8250000000000002},"last":0.67900000000000005}]},{"osd":8,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":2.556,"5min":2.556,"15min":2.556},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":2.73,"5min":2.73,"15min":2.73},"last":0.70499999999999996}]},{"osd":9,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":7.8499999999999996,"5min":7.8499999999999996,"15min":7.8499999999999996},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.5840000000000001,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":0.72699999999999998}]}]},{"osd":1,"up_from":0,"seq":21,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5880,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851720,"statfs":{"total":107374182400,"available":107368161280,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.512}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.673}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7370000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.802}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3799999999999999}]}]},{"osd":0,"up_from":0,"seq":20,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5944,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":104851656,"statfs":{"total":107374182400,"available":107368095744,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.216}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]}]},{"osd":2,"up_from":47,"seq":201863462932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":5868,"kb_used_data":612,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851732,"statfs":{"total":107374182400,"available":107368173568,"internally_reserved":0,"allocated":626688,"data_stored":295889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":3,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.52600000000000002}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.51900000000000002}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.55300000000000005}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.393,"5min":1.393,"15min":1.393},"last":0.50900000000000001}]},{"osd":7,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.53500000000000003}]},{"osd":8,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.42299999999999999}]},{"osd":9,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.5},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":0.71799999999999997}]}]},{"osd":3,"up_from":47,"seq":201863462932,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5944,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":104851656,"statfs":{"total":107374182400,"available":107368095744,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":2,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.9319999999999999,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.1960000000000002,"5min":2.1960000000000002,"15min":2.1960000000000002},"last":0.46600000000000003}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.8979999999999999,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":0.55200000000000005}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.492,"5min":3.492,"15min":3.492},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":3.3340000000000001,"5min":3.3340000000000001,"15min":3.3340000000000001},"last":0.38100000000000001}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.226,"5min":2.226,"15min":2.226},"last":0.42499999999999999}]},{"osd":7,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.50700000000000001}]},{"osd":8,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":2.359,"5min":2.359,"15min":2.359},"last":0.443},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.53200000000000003}]},{"osd":9,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":2.839,"5min":2.839,"15min":2.839},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":0.47299999999999998}]}]},{"osd":4,"up_from":47,"seq":201863462932,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5880,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851720,"statfs":{"total":107374182400,"available":107368161280,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":2,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.3119999999999998,"5min":3.3119999999999998,"15min":3.3119999999999998},"last":0.40400000000000003}]},{"osd":3,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.82699999999999996}]},{"osd":5,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":5.835,"5min":5.835,"15min":5.835},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.66000000000000003}]},{"osd":6,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":2.613,"5min":2.613,"15min":2.613},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.4470000000000001,"5min":3.4470000000000001,"15min":3.4470000000000001},"last":0.64800000000000002}]},{"osd":7,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.649,"5min":1.649,"15min":1.649},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":0.68899999999999995}]},{"osd":8,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":3.117,"5min":3.117,"15min":3.117},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"last":0.85699999999999998}]},{"osd":9,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":3.1480000000000001,"5min":3.1480000000000001,"15min":3.1480000000000001},"last":0.52800000000000002}]}]},{"osd":5,"up_from":47,"seq":201863462933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5880,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851720,"statfs":{"total":107374182400,"available":107368161280,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":2,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.36099999999999999}]},{"osd":3,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.40999999999999998}]},{"osd":4,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.55600000000000005}]},{"osd":6,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.42199999999999999}]},{"osd":7,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.44800000000000001}]},{"osd":8,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.495},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.58799999999999997}]},{"osd":9,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.53900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-25T05:44:02.959 INFO:tasks.ceph.ceph_manager.ceph:PG 1.2 is not active+clean 2023-01-25T05:44:02.960 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.2', 'version': "0'0", 'reported_seq': 66, 'reported_epoch': 56, 'state': 'active+undersized', 'last_fresh': '2023-01-25T05:43:58.608440+0000', 'last_change': '2023-01-25T05:43:58.608406+0000', 'last_active': '2023-01-25T05:43:58.608440+0000', 'last_peered': '2023-01-25T05:43:58.608440+0000', 'last_clean': '2023-01-25T05:43:55.402337+0000', 'last_became_active': '2023-01-25T05:43:58.608406+0000', 'last_became_peered': '2023-01-25T05:43:58.608406+0000', 'last_unstale': '2023-01-25T05:43:58.608440+0000', 'last_undegraded': '2023-01-25T05:43:58.608440+0000', 'last_fullsized': '2023-01-25T05:43:58.608057+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-01-25T05:42:33.343753+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-01-25T05:42:33.343753+0000', 'last_clean_scrub_stamp': '2023-01-25T05:42:33.343753+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-01-26T08:55:52.504843+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-01-25T05:44:02.960 INFO:tasks.ceph.ceph_manager.ceph:PG 1.3 is not active+clean 2023-01-25T05:44:02.961 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.3', 'version': "0'0", 'reported_seq': 70, 'reported_epoch': 56, 'state': 'active+undersized', 'last_fresh': '2023-01-25T05:43:58.607256+0000', 'last_change': '2023-01-25T05:43:58.607191+0000', 'last_active': '2023-01-25T05:43:58.607256+0000', 'last_peered': '2023-01-25T05:43:58.607256+0000', 'last_clean': '2023-01-25T05:43:55.402545+0000', 'last_became_active': '2023-01-25T05:43:58.607191+0000', 'last_became_peered': '2023-01-25T05:43:58.607191+0000', 'last_unstale': '2023-01-25T05:43:58.607256+0000', 'last_undegraded': '2023-01-25T05:43:58.607256+0000', 'last_fullsized': '2023-01-25T05:43:58.605244+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-01-25T05:42:33.343753+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-01-25T05:42:33.343753+0000', 'last_clean_scrub_stamp': '2023-01-25T05:42:33.343753+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-01-26T09:57:00.276783+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-01-25T05:44:02.961 INFO:tasks.ceph.ceph_manager.ceph:PG 1.4 is not active+clean 2023-01-25T05:44:02.961 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.4', 'version': "0'0", 'reported_seq': 46, 'reported_epoch': 56, 'state': 'active+undersized', 'last_fresh': '2023-01-25T05:43:59.516475+0000', 'last_change': '2023-01-25T05:43:59.516475+0000', 'last_active': '2023-01-25T05:43:59.516475+0000', 'last_peered': '2023-01-25T05:43:59.516475+0000', 'last_clean': '2023-01-25T05:43:43.709021+0000', 'last_became_active': '2023-01-25T05:43:57.599721+0000', 'last_became_peered': '2023-01-25T05:43:57.599721+0000', 'last_unstale': '2023-01-25T05:43:59.516475+0000', 'last_undegraded': '2023-01-25T05:43:59.516475+0000', 'last_fullsized': '2023-01-25T05:43:57.599150+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-01-25T05:42:33.343753+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-01-25T05:42:33.343753+0000', 'last_clean_scrub_stamp': '2023-01-25T05:42:33.343753+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-01-26T14:03:23.649310+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6], 'acting': [6], 'avail_no_missing': ['6'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-01-25T05:44:02.961 INFO:tasks.ceph.ceph_manager.ceph:PG 1.5 is not active+clean 2023-01-25T05:44:02.962 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.5', 'version': "0'0", 'reported_seq': 86, 'reported_epoch': 56, 'state': 'active+undersized', 'last_fresh': '2023-01-25T05:43:58.608362+0000', 'last_change': '2023-01-25T05:43:58.608316+0000', 'last_active': '2023-01-25T05:43:58.608362+0000', 'last_peered': '2023-01-25T05:43:58.608362+0000', 'last_clean': '2023-01-25T05:43:55.048820+0000', 'last_became_active': '2023-01-25T05:43:58.608316+0000', 'last_became_peered': '2023-01-25T05:43:58.608316+0000', 'last_unstale': '2023-01-25T05:43:58.608362+0000', 'last_undegraded': '2023-01-25T05:43:58.608362+0000', 'last_fullsized': '2023-01-25T05:43:58.607892+0000', 'mapping_epoch': 54, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 48, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-01-25T05:42:33.343753+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-01-25T05:42:33.343753+0000', 'last_clean_scrub_stamp': '2023-01-25T05:42:33.343753+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-01-26T09:10:36.419019+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [9], 'acting': [9], 'avail_no_missing': ['9'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 9, 'acting_primary': 9, 'purged_snaps': []} 2023-01-25T05:44:02.962 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-01-25T05:44:03.353 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:44:03.353 INFO:teuthology.orchestra.run.smithi203.stdout:{"fsid":"5b00363d-2783-4934-a70f-2b1720d57fe5","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noout flag(s) set","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":103,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":59,"num_osds":10,"num_up_osds":10,"osd_up_since":1674625440,"num_in_osds":10,"osd_in_since":1674625343,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+undersized","count":4}],"num_pgs":8,"num_pools":1,"num_objects":0,"data_bytes":0,"bytes_used":60252160,"bytes_avail":1073681571840,"bytes_total":1073741824000},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2023-01-25T05:43:32.102521+0000","services":{}},"progress_events":{}} 2023-01-25T05:44:03.367 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2023-01-25T05:44:03.368 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-25T05:44:03.683 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:44:03.685 INFO:teuthology.orchestra.run.smithi203.stderr:dumped all 2023-01-25T05:44:03.697 INFO:teuthology.orchestra.run.smithi203.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2023-01-25T05:44:02.107123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":58840,"kb_used_data":6216,"kb_used_omap":0,"kb_used_meta":52544,"kb_avail":1048517160,"statfs":{"total":1073741824000,"available":1073681571840,"internally_reserved":0,"allocated":6365184,"data_stored":3020570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53805056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":45,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-01-25T05:43:57.845120+0000","last_change":"2023-01-25T05:43:49.550719+0000","last_active":"2023-01-25T05:43:57.845120+0000","last_peered":"2023-01-25T05:43:57.845120+0000","last_clean":"2023-01-25T05:43:57.845120+0000","last_became_active":"2023-01-25T05:43:49.550449+0000","last_became_peered":"2023-01-25T05:43:49.550449+0000","last_unstale":"2023-01-25T05:43:57.845120+0000","last_undegraded":"2023-01-25T05:43:57.845120+0000","last_fullsized":"2023-01-25T05:43:57.845120+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T06:44:14.717000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":54,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-01-25T05:43:57.599466+0000","last_change":"2023-01-25T05:43:49.546867+0000","last_active":"2023-01-25T05:43:57.599466+0000","last_peered":"2023-01-25T05:43:57.599466+0000","last_clean":"2023-01-25T05:43:57.599466+0000","last_became_active":"2023-01-25T05:43:49.546602+0000","last_became_peered":"2023-01-25T05:43:49.546602+0000","last_unstale":"2023-01-25T05:43:57.599466+0000","last_undegraded":"2023-01-25T05:43:57.599466+0000","last_fullsized":"2023-01-25T05:43:57.599466+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T15:50:01.461764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":45,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-01-25T05:43:59.516529+0000","last_change":"2023-01-25T05:43:49.548900+0000","last_active":"2023-01-25T05:43:59.516529+0000","last_peered":"2023-01-25T05:43:59.516529+0000","last_clean":"2023-01-25T05:43:59.516529+0000","last_became_active":"2023-01-25T05:43:49.548607+0000","last_became_peered":"2023-01-25T05:43:49.548607+0000","last_unstale":"2023-01-25T05:43:59.516529+0000","last_undegraded":"2023-01-25T05:43:59.516529+0000","last_fullsized":"2023-01-25T05:43:59.516529+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T13:10:06.551804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":65,"reported_epoch":55,"state":"active+clean","last_fresh":"2023-01-25T05:43:59.240067+0000","last_change":"2023-01-25T05:43:49.546727+0000","last_active":"2023-01-25T05:43:59.240067+0000","last_peered":"2023-01-25T05:43:59.240067+0000","last_clean":"2023-01-25T05:43:59.240067+0000","last_became_active":"2023-01-25T05:43:49.546464+0000","last_became_peered":"2023-01-25T05:43:49.546464+0000","last_unstale":"2023-01-25T05:43:59.240067+0000","last_undegraded":"2023-01-25T05:43:59.240067+0000","last_fullsized":"2023-01-25T05:43:59.240067+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T10:57:55.038293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":66,"reported_epoch":56,"state":"active+undersized","last_fresh":"2023-01-25T05:43:58.608440+0000","last_change":"2023-01-25T05:43:58.608406+0000","last_active":"2023-01-25T05:43:58.608440+0000","last_peered":"2023-01-25T05:43:58.608440+0000","last_clean":"2023-01-25T05:43:55.402337+0000","last_became_active":"2023-01-25T05:43:58.608406+0000","last_became_peered":"2023-01-25T05:43:58.608406+0000","last_unstale":"2023-01-25T05:43:58.608440+0000","last_undegraded":"2023-01-25T05:43:58.608440+0000","last_fullsized":"2023-01-25T05:43:58.608057+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T08:55:52.504843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":70,"reported_epoch":56,"state":"active+undersized","last_fresh":"2023-01-25T05:43:58.607256+0000","last_change":"2023-01-25T05:43:58.607191+0000","last_active":"2023-01-25T05:43:58.607256+0000","last_peered":"2023-01-25T05:43:58.607256+0000","last_clean":"2023-01-25T05:43:55.402545+0000","last_became_active":"2023-01-25T05:43:58.607191+0000","last_became_peered":"2023-01-25T05:43:58.607191+0000","last_unstale":"2023-01-25T05:43:58.607256+0000","last_undegraded":"2023-01-25T05:43:58.607256+0000","last_fullsized":"2023-01-25T05:43:58.605244+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T09:57:00.276783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":46,"reported_epoch":56,"state":"active+undersized","last_fresh":"2023-01-25T05:43:59.516475+0000","last_change":"2023-01-25T05:43:59.516475+0000","last_active":"2023-01-25T05:43:59.516475+0000","last_peered":"2023-01-25T05:43:59.516475+0000","last_clean":"2023-01-25T05:43:43.709021+0000","last_became_active":"2023-01-25T05:43:57.599721+0000","last_became_peered":"2023-01-25T05:43:57.599721+0000","last_unstale":"2023-01-25T05:43:59.516475+0000","last_undegraded":"2023-01-25T05:43:59.516475+0000","last_fullsized":"2023-01-25T05:43:57.599150+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T14:03:23.649310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6],"acting":[6],"avail_no_missing":["6"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":86,"reported_epoch":56,"state":"active+undersized","last_fresh":"2023-01-25T05:43:58.608362+0000","last_change":"2023-01-25T05:43:58.608316+0000","last_active":"2023-01-25T05:43:58.608362+0000","last_peered":"2023-01-25T05:43:58.608362+0000","last_clean":"2023-01-25T05:43:55.048820+0000","last_became_active":"2023-01-25T05:43:58.608316+0000","last_became_peered":"2023-01-25T05:43:58.608316+0000","last_unstale":"2023-01-25T05:43:58.608362+0000","last_undegraded":"2023-01-25T05:43:58.608362+0000","last_fullsized":"2023-01-25T05:43:58.607892+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T09:10:36.419019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9],"acting":[9],"avail_no_missing":["9"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":39,"seq":167503724564,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5880,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851720,"statfs":{"total":107374182400,"available":107368161280,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":2,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.62},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.42199999999999999}]},{"osd":3,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.57299999999999995}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.502},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.67800000000000005}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.64000000000000001}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.78000000000000003}]},{"osd":7,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.73299999999999998}]},{"osd":8,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.754}]}]},{"osd":8,"up_from":47,"seq":201863462932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":5804,"kb_used_data":612,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":104851796,"statfs":{"total":107374182400,"available":107368239104,"internally_reserved":0,"allocated":626688,"data_stored":295889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":2,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.313},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.45100000000000001}]},{"osd":3,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.5},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.46000000000000002}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.43099999999999999}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.40600000000000003}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.51400000000000001}]},{"osd":7,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.52600000000000002}]},{"osd":9,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.56000000000000005}]}]},{"osd":7,"up_from":47,"seq":201863462933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5880,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851720,"statfs":{"total":107374182400,"available":107368161280,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":2,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.30599999999999999}]},{"osd":3,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.38400000000000001}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.42399999999999999}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.44}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.58499999999999996}]},{"osd":8,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.505},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.55800000000000005}]},{"osd":9,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.47399999999999998}]}]},{"osd":6,"up_from":47,"seq":201863462932,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5880,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851720,"statfs":{"total":107374182400,"available":107368161280,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":2,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":7.7309999999999999,"5min":7.7309999999999999,"15min":7.7309999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":2.5139999999999998,"5min":2.5139999999999998,"15min":2.5139999999999998},"last":0.56599999999999995}]},{"osd":3,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":2.7549999999999999,"5min":2.7549999999999999,"15min":2.7549999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":7.7729999999999997,"5min":7.7729999999999997,"15min":7.7729999999999997},"last":0.63300000000000001}]},{"osd":4,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":2.5350000000000001,"5min":2.5350000000000001,"15min":2.5350000000000001},"last":0.496},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.7799999999999998,"5min":2.7799999999999998,"15min":2.7799999999999998},"last":0.621}]},{"osd":5,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":7.7990000000000004,"5min":7.7990000000000004,"15min":7.7990000000000004},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.4569999999999999,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":0.64900000000000002}]},{"osd":7,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":7.8250000000000002,"5min":7.8250000000000002,"15min":7.8250000000000002},"last":0.67900000000000005}]},{"osd":8,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":2.556,"5min":2.556,"15min":2.556},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":2.73,"5min":2.73,"15min":2.73},"last":0.70499999999999996}]},{"osd":9,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":7.8499999999999996,"5min":7.8499999999999996,"15min":7.8499999999999996},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.5840000000000001,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":0.72699999999999998}]}]},{"osd":1,"up_from":0,"seq":21,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5880,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851720,"statfs":{"total":107374182400,"available":107368161280,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.512}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.673}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7370000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.802}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3799999999999999}]}]},{"osd":0,"up_from":0,"seq":20,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5944,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":104851656,"statfs":{"total":107374182400,"available":107368095744,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.216}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32700000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]}]},{"osd":2,"up_from":47,"seq":201863462932,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":5868,"kb_used_data":612,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851732,"statfs":{"total":107374182400,"available":107368173568,"internally_reserved":0,"allocated":626688,"data_stored":295889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":3,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.52600000000000002}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.51900000000000002}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.55300000000000005}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.393,"5min":1.393,"15min":1.393},"last":0.50900000000000001}]},{"osd":7,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.53500000000000003}]},{"osd":8,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.42299999999999999}]},{"osd":9,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.5},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":0.71799999999999997}]}]},{"osd":3,"up_from":47,"seq":201863462932,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5944,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":104851656,"statfs":{"total":107374182400,"available":107368095744,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":2,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.9319999999999999,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.1960000000000002,"5min":2.1960000000000002,"15min":2.1960000000000002},"last":0.46600000000000003}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.8979999999999999,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":0.55200000000000005}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.492,"5min":3.492,"15min":3.492},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":3.3340000000000001,"5min":3.3340000000000001,"15min":3.3340000000000001},"last":0.38100000000000001}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.226,"5min":2.226,"15min":2.226},"last":0.42499999999999999}]},{"osd":7,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.50700000000000001}]},{"osd":8,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":2.359,"5min":2.359,"15min":2.359},"last":0.443},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.53200000000000003}]},{"osd":9,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":2.839,"5min":2.839,"15min":2.839},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":0.47299999999999998}]}]},{"osd":4,"up_from":47,"seq":201863462932,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5880,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851720,"statfs":{"total":107374182400,"available":107368161280,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":2,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.3119999999999998,"5min":3.3119999999999998,"15min":3.3119999999999998},"last":0.40400000000000003}]},{"osd":3,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.82699999999999996}]},{"osd":5,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":5.835,"5min":5.835,"15min":5.835},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.66000000000000003}]},{"osd":6,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":2.613,"5min":2.613,"15min":2.613},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.4470000000000001,"5min":3.4470000000000001,"15min":3.4470000000000001},"last":0.64800000000000002}]},{"osd":7,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.649,"5min":1.649,"15min":1.649},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":0.68899999999999995}]},{"osd":8,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":3.117,"5min":3.117,"15min":3.117},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"last":0.85699999999999998}]},{"osd":9,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":3.1480000000000001,"5min":3.1480000000000001,"15min":3.1480000000000001},"last":0.52800000000000002}]}]},{"osd":5,"up_from":47,"seq":201863462933,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5880,"kb_used_data":624,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851720,"statfs":{"total":107374182400,"available":107368161280,"internally_reserved":0,"allocated":638976,"data_stored":303599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":2,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.36099999999999999}]},{"osd":3,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.40999999999999998}]},{"osd":4,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.55600000000000005}]},{"osd":6,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.42199999999999999}]},{"osd":7,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.44800000000000001}]},{"osd":8,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.495},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.58799999999999997}]},{"osd":9,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.53900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-25T05:44:06.700 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-25T05:44:07.027 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:44:07.028 INFO:teuthology.orchestra.run.smithi203.stderr:dumped all 2023-01-25T05:44:07.040 INFO:teuthology.orchestra.run.smithi203.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2023-01-25T05:44:06.107730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":59480,"kb_used_data":6600,"kb_used_omap":0,"kb_used_meta":52800,"kb_avail":1048516520,"statfs":{"total":1073741824000,"available":1073680916480,"internally_reserved":0,"allocated":6758400,"data_stored":3273790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54067200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":47,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-01-25T05:44:01.078714+0000","last_change":"2023-01-25T05:43:49.550719+0000","last_active":"2023-01-25T05:44:01.078714+0000","last_peered":"2023-01-25T05:44:01.078714+0000","last_clean":"2023-01-25T05:44:01.078714+0000","last_became_active":"2023-01-25T05:43:49.550449+0000","last_became_peered":"2023-01-25T05:43:49.550449+0000","last_unstale":"2023-01-25T05:44:01.078714+0000","last_undegraded":"2023-01-25T05:44:01.078714+0000","last_fullsized":"2023-01-25T05:44:01.078714+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T06:44:14.717000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":56,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-01-25T05:44:01.079097+0000","last_change":"2023-01-25T05:43:49.546867+0000","last_active":"2023-01-25T05:44:01.079097+0000","last_peered":"2023-01-25T05:44:01.079097+0000","last_clean":"2023-01-25T05:44:01.079097+0000","last_became_active":"2023-01-25T05:43:49.546602+0000","last_became_peered":"2023-01-25T05:43:49.546602+0000","last_unstale":"2023-01-25T05:44:01.079097+0000","last_undegraded":"2023-01-25T05:44:01.079097+0000","last_fullsized":"2023-01-25T05:44:01.079097+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T15:50:01.461764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":46,"reported_epoch":57,"state":"active+clean","last_fresh":"2023-01-25T05:44:00.152747+0000","last_change":"2023-01-25T05:43:49.548900+0000","last_active":"2023-01-25T05:44:00.152747+0000","last_peered":"2023-01-25T05:44:00.152747+0000","last_clean":"2023-01-25T05:44:00.152747+0000","last_became_active":"2023-01-25T05:43:49.548607+0000","last_became_peered":"2023-01-25T05:43:49.548607+0000","last_unstale":"2023-01-25T05:44:00.152747+0000","last_undegraded":"2023-01-25T05:44:00.152747+0000","last_fullsized":"2023-01-25T05:44:00.152747+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T13:10:06.551804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":66,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-01-25T05:43:59.516415+0000","last_change":"2023-01-25T05:43:49.546727+0000","last_active":"2023-01-25T05:43:59.516415+0000","last_peered":"2023-01-25T05:43:59.516415+0000","last_clean":"2023-01-25T05:43:59.516415+0000","last_became_active":"2023-01-25T05:43:49.546464+0000","last_became_peered":"2023-01-25T05:43:49.546464+0000","last_unstale":"2023-01-25T05:43:59.516415+0000","last_undegraded":"2023-01-25T05:43:59.516415+0000","last_fullsized":"2023-01-25T05:43:59.516415+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T10:57:55.038293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":78,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-01-25T05:44:01.613444+0000","last_change":"2023-01-25T05:44:01.613444+0000","last_active":"2023-01-25T05:44:01.613444+0000","last_peered":"2023-01-25T05:44:01.613444+0000","last_clean":"2023-01-25T05:44:01.613444+0000","last_became_active":"2023-01-25T05:44:01.613204+0000","last_became_peered":"2023-01-25T05:44:01.613204+0000","last_unstale":"2023-01-25T05:44:01.613444+0000","last_undegraded":"2023-01-25T05:44:01.613444+0000","last_fullsized":"2023-01-25T05:44:01.613444+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T09:57:00.276783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":82,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-01-25T05:44:01.613722+0000","last_change":"2023-01-25T05:44:01.613722+0000","last_active":"2023-01-25T05:44:01.613722+0000","last_peered":"2023-01-25T05:44:01.613722+0000","last_clean":"2023-01-25T05:44:01.613722+0000","last_became_active":"2023-01-25T05:44:01.613464+0000","last_became_peered":"2023-01-25T05:44:01.613464+0000","last_unstale":"2023-01-25T05:44:01.613722+0000","last_undegraded":"2023-01-25T05:44:01.613722+0000","last_fullsized":"2023-01-25T05:44:01.613722+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T08:52:56.950334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-01-25T05:44:01.948246+0000","last_change":"2023-01-25T05:44:01.948246+0000","last_active":"2023-01-25T05:44:01.948246+0000","last_peered":"2023-01-25T05:44:01.948246+0000","last_clean":"2023-01-25T05:44:01.948246+0000","last_became_active":"2023-01-25T05:44:01.947953+0000","last_became_peered":"2023-01-25T05:44:01.947953+0000","last_unstale":"2023-01-25T05:44:01.948246+0000","last_undegraded":"2023-01-25T05:44:01.948246+0000","last_fullsized":"2023-01-25T05:44:01.948246+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T16:35:45.990734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":99,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-01-25T05:44:01.848789+0000","last_change":"2023-01-25T05:44:01.848789+0000","last_active":"2023-01-25T05:44:01.848789+0000","last_peered":"2023-01-25T05:44:01.848789+0000","last_clean":"2023-01-25T05:44:01.848789+0000","last_became_active":"2023-01-25T05:44:01.848541+0000","last_became_peered":"2023-01-25T05:44:01.848541+0000","last_unstale":"2023-01-25T05:44:01.848789+0000","last_undegraded":"2023-01-25T05:44:01.848789+0000","last_fullsized":"2023-01-25T05:44:01.848789+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T06:47:47.463060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":39,"seq":167503724565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5980,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":104851620,"statfs":{"total":107374182400,"available":107368058880,"internally_reserved":0,"allocated":675840,"data_stored":327379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":2,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.48899999999999999}]},{"osd":3,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.51200000000000001}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.27800000000000002}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.45000000000000001}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.46700000000000003}]},{"osd":7,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.56599999999999995}]},{"osd":8,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.497}]}]},{"osd":8,"up_from":47,"seq":201863462933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":5916,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851684,"statfs":{"total":107374182400,"available":107368124416,"internally_reserved":0,"allocated":675840,"data_stored":327379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":2,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.41899999999999998}]},{"osd":3,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.372},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.38100000000000001}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.48699999999999999}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.495}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.53000000000000003}]},{"osd":7,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.57299999999999995}]},{"osd":9,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.53900000000000003}]}]},{"osd":7,"up_from":47,"seq":201863462934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5916,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851684,"statfs":{"total":107374182400,"available":107368124416,"internally_reserved":0,"allocated":675840,"data_stored":327379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":2,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.30599999999999999}]},{"osd":3,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.38400000000000001}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.42399999999999999}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.44}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.58499999999999996}]},{"osd":8,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.505},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.55800000000000005}]},{"osd":9,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.47399999999999998}]}]},{"osd":6,"up_from":47,"seq":201863462933,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5916,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851684,"statfs":{"total":107374182400,"available":107368124416,"internally_reserved":0,"allocated":675840,"data_stored":327379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":2,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":7.7309999999999999,"5min":7.7309999999999999,"15min":7.7309999999999999},"last":0.372},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":2.5139999999999998,"5min":2.5139999999999998,"15min":2.5139999999999998},"last":0.41999999999999998}]},{"osd":3,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":2.7549999999999999,"5min":2.7549999999999999,"15min":2.7549999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":7.7729999999999997,"5min":7.7729999999999997,"15min":7.7729999999999997},"last":0.54100000000000004}]},{"osd":4,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":2.5350000000000001,"5min":2.5350000000000001,"15min":2.5350000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.7799999999999998,"5min":2.7799999999999998,"15min":2.7799999999999998},"last":0.60899999999999999}]},{"osd":5,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":7.7990000000000004,"5min":7.7990000000000004,"15min":7.7990000000000004},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.4569999999999999,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":0.63700000000000001}]},{"osd":7,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":7.8250000000000002,"5min":7.8250000000000002,"15min":7.8250000000000002},"last":0.71299999999999997}]},{"osd":8,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":2.556,"5min":2.556,"15min":2.556},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":2.73,"5min":2.73,"15min":2.73},"last":0.625}]},{"osd":9,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":7.8499999999999996,"5min":7.8499999999999996,"15min":7.8499999999999996},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.5840000000000001,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":0.59599999999999997}]}]},{"osd":1,"up_from":58,"seq":249108103190,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5980,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":104851620,"statfs":{"total":107374182400,"available":107368058880,"internally_reserved":0,"allocated":675840,"data_stored":327379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]}]},{"osd":0,"up_from":58,"seq":249108103189,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5980,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":104851620,"statfs":{"total":107374182400,"available":107368058880,"internally_reserved":0,"allocated":675840,"data_stored":327379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[1,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]},{"osd":2,"up_from":47,"seq":201863462933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":5916,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851684,"statfs":{"total":107374182400,"available":107368124416,"internally_reserved":0,"allocated":675840,"data_stored":327379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":3,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.47099999999999997}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.28499999999999998}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.496},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.45700000000000002}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.393,"5min":1.393,"15min":1.393},"last":0.53700000000000003}]},{"osd":7,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.56200000000000006}]},{"osd":8,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.66900000000000004}]},{"osd":9,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":0.58399999999999996}]}]},{"osd":3,"up_from":47,"seq":201863462933,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5980,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":104851620,"statfs":{"total":107374182400,"available":107368058880,"internally_reserved":0,"allocated":675840,"data_stored":327379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]},{"osd":2,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.9319999999999999,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.1960000000000002,"5min":2.1960000000000002,"15min":2.1960000000000002},"last":0.55500000000000005}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.8979999999999999,"5min":1.8979999999999999,"15min":1.8979999999999999},"last":0.57899999999999996}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.492,"5min":3.492,"15min":3.492},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":3.3340000000000001,"5min":3.3340000000000001,"15min":3.3340000000000001},"last":0.504}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.226,"5min":2.226,"15min":2.226},"last":0.40300000000000002}]},{"osd":7,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.88100000000000001}]},{"osd":8,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":2.359,"5min":2.359,"15min":2.359},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.76100000000000001}]},{"osd":9,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":2.839,"5min":2.839,"15min":2.839},"last":0.63},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":0.92700000000000005}]}]},{"osd":4,"up_from":47,"seq":201863462933,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5980,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":104851620,"statfs":{"total":107374182400,"available":107368058880,"internally_reserved":0,"allocated":675840,"data_stored":327379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]},{"osd":2,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.371},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.3119999999999998,"5min":3.3119999999999998,"15min":3.3119999999999998},"last":0.45500000000000002}]},{"osd":3,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.628}]},{"osd":5,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":5.835,"5min":5.835,"15min":5.835},"last":0.504},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.52100000000000002}]},{"osd":6,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":2.613,"5min":2.613,"15min":2.613},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.4470000000000001,"5min":3.4470000000000001,"15min":3.4470000000000001},"last":0.67000000000000004}]},{"osd":7,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.649,"5min":1.649,"15min":1.649},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":0.58899999999999997}]},{"osd":8,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":3.117,"5min":3.117,"15min":3.117},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"last":0.66100000000000003}]},{"osd":9,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":3.1480000000000001,"5min":3.1480000000000001,"15min":3.1480000000000001},"last":0.69699999999999995}]}]},{"osd":5,"up_from":47,"seq":201863462934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5916,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851684,"statfs":{"total":107374182400,"available":107368124416,"internally_reserved":0,"allocated":675840,"data_stored":327379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25}]},{"osd":2,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.54400000000000004}]},{"osd":3,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.52000000000000002}]},{"osd":4,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.50900000000000001}]},{"osd":6,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.5}]},{"osd":7,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.44},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.47499999999999998}]},{"osd":8,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.48299999999999998}]},{"osd":9,"last update":"Wed Jan 25 05:43:30 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.55500000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-01-25T05:44:07.042 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-01-25T05:44:07.043 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-25T05:44:07.356 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-01-25T05:44:07.358 INFO:teuthology.orchestra.run.smithi203.stderr:dumped all 2023-01-25T05:44:07.369 INFO:teuthology.orchestra.run.smithi203.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2023-01-25T05:44:06.107730+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":8,"kb":1048576000,"kb_used":59480,"kb_used_data":6600,"kb_used_omap":0,"kb_used_meta":52800,"kb_avail":1048516520,"statfs":{"total":1073741824000,"available":1073680916480,"internally_reserved":0,"allocated":6758400,"data_stored":3273790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54067200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":47,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-01-25T05:44:01.078714+0000","last_change":"2023-01-25T05:43:49.550719+0000","last_active":"2023-01-25T05:44:01.078714+0000","last_peered":"2023-01-25T05:44:01.078714+0000","last_clean":"2023-01-25T05:44:01.078714+0000","last_became_active":"2023-01-25T05:43:49.550449+0000","last_became_peered":"2023-01-25T05:43:49.550449+0000","last_unstale":"2023-01-25T05:44:01.078714+0000","last_undegraded":"2023-01-25T05:44:01.078714+0000","last_fullsized":"2023-01-25T05:44:01.078714+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T06:44:14.717000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":56,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-01-25T05:44:01.079097+0000","last_change":"2023-01-25T05:43:49.546867+0000","last_active":"2023-01-25T05:44:01.079097+0000","last_peered":"2023-01-25T05:44:01.079097+0000","last_clean":"2023-01-25T05:44:01.079097+0000","last_became_active":"2023-01-25T05:43:49.546602+0000","last_became_peered":"2023-01-25T05:43:49.546602+0000","last_unstale":"2023-01-25T05:44:01.079097+0000","last_undegraded":"2023-01-25T05:44:01.079097+0000","last_fullsized":"2023-01-25T05:44:01.079097+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T15:50:01.461764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":46,"reported_epoch":57,"state":"active+clean","last_fresh":"2023-01-25T05:44:00.152747+0000","last_change":"2023-01-25T05:43:49.548900+0000","last_active":"2023-01-25T05:44:00.152747+0000","last_peered":"2023-01-25T05:44:00.152747+0000","last_clean":"2023-01-25T05:44:00.152747+0000","last_became_active":"2023-01-25T05:43:49.548607+0000","last_became_peered":"2023-01-25T05:43:49.548607+0000","last_unstale":"2023-01-25T05:44:00.152747+0000","last_undegraded":"2023-01-25T05:44:00.152747+0000","last_fullsized":"2023-01-25T05:44:00.152747+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T13:10:06.551804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":66,"reported_epoch":56,"state":"active+clean","last_fresh":"2023-01-25T05:43:59.516415+0000","last_change":"2023-01-25T05:43:49.546727+0000","last_active":"2023-01-25T05:43:59.516415+0000","last_peered":"2023-01-25T05:43:59.516415+0000","last_clean":"2023-01-25T05:43:59.516415+0000","last_became_active":"2023-01-25T05:43:49.546464+0000","last_became_peered":"2023-01-25T05:43:49.546464+0000","last_unstale":"2023-01-25T05:43:59.516415+0000","last_undegraded":"2023-01-25T05:43:59.516415+0000","last_fullsized":"2023-01-25T05:43:59.516415+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T10:57:55.038293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9],"acting":[5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":78,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-01-25T05:44:01.613444+0000","last_change":"2023-01-25T05:44:01.613444+0000","last_active":"2023-01-25T05:44:01.613444+0000","last_peered":"2023-01-25T05:44:01.613444+0000","last_clean":"2023-01-25T05:44:01.613444+0000","last_became_active":"2023-01-25T05:44:01.613204+0000","last_became_peered":"2023-01-25T05:44:01.613204+0000","last_unstale":"2023-01-25T05:44:01.613444+0000","last_undegraded":"2023-01-25T05:44:01.613444+0000","last_fullsized":"2023-01-25T05:44:01.613444+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T09:57:00.276783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":82,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-01-25T05:44:01.613722+0000","last_change":"2023-01-25T05:44:01.613722+0000","last_active":"2023-01-25T05:44:01.613722+0000","last_peered":"2023-01-25T05:44:01.613722+0000","last_clean":"2023-01-25T05:44:01.613722+0000","last_became_active":"2023-01-25T05:44:01.613464+0000","last_became_peered":"2023-01-25T05:44:01.613464+0000","last_unstale":"2023-01-25T05:44:01.613722+0000","last_undegraded":"2023-01-25T05:44:01.613722+0000","last_fullsized":"2023-01-25T05:44:01.613722+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T08:52:56.950334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":61,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-01-25T05:44:01.948246+0000","last_change":"2023-01-25T05:44:01.948246+0000","last_active":"2023-01-25T05:44:01.948246+0000","last_peered":"2023-01-25T05:44:01.948246+0000","last_clean":"2023-01-25T05:44:01.948246+0000","last_became_active":"2023-01-25T05:44:01.947953+0000","last_became_peered":"2023-01-25T05:44:01.947953+0000","last_unstale":"2023-01-25T05:44:01.948246+0000","last_undegraded":"2023-01-25T05:44:01.948246+0000","last_fullsized":"2023-01-25T05:44:01.948246+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T16:35:45.990734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":99,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-01-25T05:44:01.848789+0000","last_change":"2023-01-25T05:44:01.848789+0000","last_active":"2023-01-25T05:44:01.848789+0000","last_peered":"2023-01-25T05:44:01.848789+0000","last_clean":"2023-01-25T05:44:01.848789+0000","last_became_active":"2023-01-25T05:44:01.848541+0000","last_became_peered":"2023-01-25T05:44:01.848541+0000","last_unstale":"2023-01-25T05:44:01.848789+0000","last_undegraded":"2023-01-25T05:44:01.848789+0000","last_fullsized":"2023-01-25T05:44:01.848789+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-25T05:42:33.343753+0000","last_clean_scrub_stamp":"2023-01-25T05:42:33.343753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-01-26T06:47:47.463060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0],"acting":[9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":9}],"osd_stats":[{"osd":9,"up_from":39,"seq":167503724565,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5980,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":104851620,"statfs":{"total":107374182400,"available":107368058880,"internally_reserved":0,"allocated":675840,"data_stored":327379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[2,3,4,5,6,7,8],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":2,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.48899999999999999}]},{"osd":3,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.51200000000000001}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.27800000000000002}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.45000000000000001}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.46700000000000003}]},{"osd":7,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.56599999999999995}]},{"osd":8,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.497}]}]},{"osd":8,"up_from":47,"seq":201863462933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":5916,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851684,"statfs":{"total":107374182400,"available":107368124416,"internally_reserved":0,"allocated":675840,"data_stored":327379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,6,7,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":2,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.41899999999999998}]},{"osd":3,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.372},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.38100000000000001}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.48699999999999999}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.495}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.381,"5min":1.381,"15min":1.381},"last":0.53000000000000003}]},{"osd":7,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.57299999999999995}]},{"osd":9,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.53900000000000003}]}]},{"osd":7,"up_from":47,"seq":201863462934,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5916,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851684,"statfs":{"total":107374182400,"available":107368124416,"internally_reserved":0,"allocated":675840,"data_stored":327379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[2,3,4,5,6,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":2,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.30599999999999999}]},{"osd":3,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.6850000000000001,"5min":1.6850000000000001,"15min":1.6850000000000001},"last":0.38400000000000001}]},{"osd":4,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.42399999999999999}]},{"osd":5,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5209999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.44}]},{"osd":6,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.58499999999999996}]},{"osd":8,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.505},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.55800000000000005}]},{"osd":9,"last update":"Wed Jan 25 05:43:32 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.47399999999999998}]}]},{"osd":6,"up_from":47,"seq":201863462933,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5916,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":104851684,"statfs":{"total":107374182400,"available":107368124416,"internally_reserved":0,"allocated":675840,"data_stored":327379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40899999999999997}]},{"osd":2,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":7.7309999999999999,"5min":7.7309999999999999,"15min":7.7309999999999999},"last":0.372},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":2.5139999999999998,"5min":2.5139999999999998,"15min":2.5139999999999998},"last":0.41999999999999998}]},{"osd":3,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":2.7549999999999999,"5min":2.7549999999999999,"15min":2.7549999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":7.7729999999999997,"5min":7.7729999999999997,"15min":7.7729999999999997},"last":0.54100000000000004}]},{"osd":4,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":2.5350000000000001,"5min":2.5350000000000001,"15min":2.5350000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.7799999999999998,"5min":2.7799999999999998,"15min":2.7799999999999998},"last":0.60899999999999999}]},{"osd":5,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":7.7990000000000004,"5min":7.7990000000000004,"15min":7.7990000000000004},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.4569999999999999,"5min":2.4569999999999999,"15min":2.4569999999999999},"last":0.63700000000000001}]},{"osd":7,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":2.8029999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":7.8250000000000002,"5min":7.8250000000000002,"15min":7.8250000000000002},"last":0.71299999999999997}]},{"osd":8,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":2.556,"5min":2.556,"15min":2.556},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":2.73,"5min":2.73,"15min":2.73},"last":0.625}]},{"osd":9,"last update":"Wed Jan 25 05:43:31 2023","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.054,"15min":1.054},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":7.8499999999999996,"5min":7.8499999999999996,"15min":7.8499999999999996},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.5840000000000001,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":0.59599999999999997}]}]},{"osd":1,"up_from":58,"seq":249108103190,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5980,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":104851620,"statfs":{"total":107374182400,"available":107368058880,"internally_reserved":0,"allocated":675840,"data_stored":327379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,2,3,4,5,6,7,8,9],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]}]},{"osd":0,"up_from":58,"seq":249108103189,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":5980,"kb_used_data":660,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":104851620,"statfs":{"total":107374182400,"available":107368058880,"internal